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FORiWORD . 



The history ot educational research evidences thfe* unending attempt to 
improve educational jDractice. Although much important information has 
been gleaned frorn such research, a problem remains. There is a quantity of 
available d^ta but too l1ttled(Bmonstriited adaptation'ofiTHractual practice. 
There is even less evidence of training strategies designed for these adap- 
tation processes.. The Teacher Corps is attempting to alter this^ imbalance. 
• ' This document-reports the firs^.year activities of Teacher Corps projects 
demonstrating the^training framework, entitled the Adaptation of Research 
Findings. As one of five»frameworks> the Research Adaptation projects are 
attempting to incorporat^i/ito the design of then- training/retraining 
programs the results of research, emphrical practices and processes which 
have proven effective and r^le^ant tocthe learning and educational processes 
foN^chools servmg low-inc;orpe populations' 

.This framework, like.th^t^fer four, is viewepi as a jesponse to.the call 
for a systematicrchange'in teacher education. SpeciffciUy, this framework 
addresses the long time need to support efforts to, <rst, adapt research 
results as needed, secondly, |o demonstrate training and retraining ap- 
propriate to adaptation, and Wirdly. to apply such results to the classroom 
*and hopefully improve educAiopal opportunities for students. More suc- 
cinctly, the framework for training to ada0t research findings demonstrates 
the effort to mejge theory with practice: 

At the sajjie time, the findings of the frameWo/k to date have verified- ' 
what in the palst has been left unsaid. Namely, much that is pubjished as 
research-based for teacher use is not useful. Many of the materials 
developed for teacher use hav^not been adequately tested, the materials 
often become obsolete by the time they reach the teacher. The extent to 
which research results have had to be adapted clearty supports this con- 
tention. ■ ; ' ^ 

The Teacher Corps projects addressing thp concerns related to iden- 
tifying and s^ectirtfe app«Mriate research results, adapting such results to 
local needs,^ndAde$i»gnJng appropriate training have spent a year developing 
and documenting the prbcesses used to achieve their reported resulS. 

This document attempts io report a composite picture of those project 
processes. An individual project (Profile has not tieh attempted. Rather, ' 
each project's documented implementation proces^s was synthesized and . 
reported under major headings which were identified by the document's 
author in conjunction with. the director^ of t|)je.proiects..The effort should 
benefit all Teacher Corps projects in addition to dlHers 'involved in in^ 



proving teacher education. 





Dr. William L. Smith • - 
Director, Teacher Corps ^ 
Uls. Office of Education^ 
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CHAPTER I . ■ . . • 

EVALUATING IMPACT — AN OVERVJEW 

Few words in our vocabulary can create more of an emotional reaction 
than the term "evaluation/' The term ^as associated in our youth with test- 
taking, grading, and decisions about our future. Our own personal worth is 
often shaped by the evaluations we s-eceive from others. As adults we try to . 
have a more objective view of the ^valuatioq process, but it always seems 
more acceptable to evaluate others than to be evaluated ourselves. Although . 
evaluations of some form or another take place every day, it Is difficult to 
becort^ accustomed to the process. 

For a number of important reasons, the evaluation process is gaining in- 
creasing significance in the field of education. Those who fund education 
are becoming concerned about the documentation of the» Impact on the 
quality of education students receive. In recent years increasing numbers of 
federally funded educational programs have included evaluation of project 
outcomes as a mandatory component of their projects. For exam pie, .Titl6 I IF 
(now IV-C) of. the Elementary and Secondary Education Act, which funds 
Innovative projects in public schools, requires local, state, and somTetimes 
federal evaluation of projects; Teacher Corps now requires evaluation as a 
component, for all its projects. The basic question asked ir^ most cases is . 
"What impajsthas this project had'upon the recipients of the services which 
were delivered?" * * 

The consequence of the recent emphasis on accountability has been to 
sharply increase the emphasis on evaluation within projects of all types. It is 
not uncommon for a proposal writer to be required to include a section on 
proposed evaluation activities and to indicate in the personnel section that 
at least a ha^f-time person with evalaation skills will be employed to carry 
out project evaluation activities. While sjmie project directors are fortunate 
enough to find an experienced ^va^tor who can assist jn the con- 
ceptualization and implementation of an ^aluatiori plan, this is. nbt:;" 
comofion. It is more-often the casalh'a^# person is hired who has )^^.o^!^i*^jf 
two statistics cour?fes and little experience in project e\/blulitffifj|j^i^^' 
(^'evaluator" may have the assistance of a trajned evaluation specialisjt'fPprri a^ 
nearby university; however, tlrfie and fundi .fbt spec^aHsts are often (imit^cL*^ , 

Because of the lack of experienced evaluat(^s, this monograph has been 
prepared to proyid^ project directors and evaluators with an overview of^the 
evaluation proces^ as Jf.applies to educational projects and how it can be 
»perform^ systematically and effectively. The mpst important characteristic 
of this monograph is-'rtot that it will provide a "textbook" solution to all real 
worrd evaluation problems, but rather, that it is aimed directly at the 
identification and consideration of problems which emerge in pcojects as 
evaluators attempt ^o (Resign, implement, and interpret an evaluation 
strategy. The monograph draws directly on the experiences of ten Teacher 
Corps project evaluators. These evaluators represent a variety of academic 
backgrounds and several different types of projects as well. It is anticipated 
that readers can identify their interests* with those of a number of ' the 
projects which are described briefly in the appendix.' 
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It is hoped the reader will benefit directly from the experiences of the 
evaluators as they implemented their evaluation plans. No attempt has^een 
made to. gloss over the difficult problems or tiS make things whictt are 
relatively easy appear more difficult. The remainder of this chapter will 
briefly describe tl>e succeeding sections, of the monograph.,.Based on this 
overvfew, the reader may wish tp refer directly to specific topics which are of 
current interest, or to read systematically from the beginning to obtain a 
total perspective on the evaluation process.. 

First, in Chapter II, an in-depth look at the evajuation process within 
various' education projects is presented, and the differences between impact 
evaluation and process evaluation are discussed. The point will be made that 
in order to effectively describe the impact of a project/ it is necessary to 
document and describe project activities as they are planned and carri^ 
out. This information is valuable not only for indicating tl^e effectiveness of 
the project and enhancing the probability that the project products and 
procedures will be useful to other ed.iiJcators, but also for making project 
management decisions relative to procedural pr program changes. A 
cQp^)arison among projects which focus either on student outcomes, 
teacher outcomes, or institutional outcomes is presented and implication's 
these^fferent foci have for the evaluation process are naade. ' * • 

Considerations of where to begin to conceptualize- the evaluation 
process are presented in Chapter III. Typical evaluation systems begin ^ith 
needs assessment activities such as the identification and prioritization of 
needs which should be tiddressed by the project. These assessments usually 
result in the statement of pl-oject goals^^hifh are converted into specific 
objectives. ^ • 

While the process of conducting needs assessments to establish project 
goafs and objectives Is fairly straightforward, tWe are^Droblems related to 
the^e activities which can ancTwill occur. These problems are related to the 
^project proposal, the collection of data, and the actual acceptance of stated 
project goals,by the participants in the project. Because of these problems, 
there are often limitations on the objectives which are established, t 

The' next chapter. Chapter ^V, discusses the de'^ign of impact evaluatipn 
studies. While, some, projects lend theniselves to classical, experimental, 
design' tech/iiques which hSve been well known for years, there are often 
extenuating cirtumstances which make the implementation of such designs 
almost impossible. Therefore, some\alternative approacf^r^s to evaluation 
studl?»*such as quasi-experimentat<lesi^ns and the establishment of criterion 
standards for evaluatiouf will Wtpj^^ented. The advantages and disad- 
vantages of these approathes^i^f b'e di^'c^ssed along with a consideration of 
^ unanticipated outcomes ap'd a^eii^t^^ evaluation. 

After an Impact evaluation d^igji has been established, the next critical- 
step, as discussed in Chapter V, isThe design and/or selection of evaluation 
instruments which will be e(nployed. It is not'unusual for a project which 
foCtrs^s on student oiftcomes to select a well-known standardized test of 
academic ability as its major impact indicator. However, there af5**numerous 
problems with this approach, and these will be discussed, fyoj^ct objec;,tives 
which relate to skill and attitudinal outcorriestio not always lend themselves 
tojstandardized testing.. Other types of assessment instruments^ need to be 
identified. A variety of types of instruments which can be used for the^e 
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purposes will be considered a3' well BS. behaVioral indicators which can be 
used to -evaluate t^roject outcomes, , 

' * It might appear that,/after the goals* and objectives of a project* have 
been established,, tJ^^ evatuBtipn. design completed, and the appropriate 
evaluation ipstrumefits located, the evaluator can^iretite to other interests 
and wait yntiUhe project is eomplete to collect data and make conclusions 
about project effectiveness." However, it is arguecj in Chapter VI that this is» 
fiot the case, but: rather *that the process of evaluating ongoing activities 
during the proji/ect4sjD^<firitical importance both to the management of the 
project and to the eventual exportability or sharing of the outcomes ot the , 
project. ProceduTel and instruments which can be used for process 
evaluation will^ be considered along with the utilit^i^. of this xiata while "ttie 
project is'under way. ^ . ' 

The need for an information system which is available to both the 
project director and other project personnel will be discussed. The pros and 
cons of a variety of approaches to establishing and maintaining such in* 
formation systems will be presented. 

The' importance of careful preparation for the presentation of data 
gathered during the evaluation process is considered in Chapter Vll. Too 
•often extensive work is dorie to collect valuable evaluation data which is 
presented in such a confusing manper that it is of little use to the consumers 
of the information. This chapter will emphasize the importance of the 
organization, display, and interpretation of data in order to maximize its. 
^.usefulness by the intended recipients. 

Chapter VITI focuses on a nurhbfr of the major problems and issues. 
Which arise in the impact evaluation process. These problems include botly*" 
organization and "people" problems and how they relate to the evaluatityi ■ 
, process. Some of the problems may appear to be trivial, however they c^n 
have a significant impact on the ability of the evaluator to collect and in- 
terpret project data. These c»re problems which, if anticipated in advance,, 
can often be minimized or elirriinated. However, if they are not anticipated^ 
they may. raise major questions about the effectiveness of the evaluation. 

The final chapter, Chapter' IX, summarizes the importance of |?oth 
process and impact evaluation and the cpnsiderable significance they play 
in terms of the educational benefits of projects. Brief attention is given to 
the political necessities of implementing evaluation and the benefits which 
can emerge through this process. Futureydirections for evaluation and the 
evaluator are also considered. ?^ 



CHAPTER II 

r R^LE AND SCOPE OF \ 
: • :^JECT EVALUATION^ 

Historical Role of Evaluation C 

Several decades ago the terms evaluation and.educational research 
were nearly synonymous. Educational research was typically performed at 
that time by individuals trained in the field of educational psychology or 
statistics. The^primary purpose of those studies was to investigate the role of 
various learning patterns and instructional techniques. The methodology 
was often a combination of procedures adopted from the parent field of 
psychology and statistical approaches borrowed from the field of' 
agriculture. Educational researchers were distinguished by their attempt to 
use meaningful fearning materials in a real or simulated classroom learning 
situation. 

In this context, evaluation was often equivalent to running T-tests be- 
tween scores for an experimental group and a controlled group to determine 
which had learned nhore or had used the lesser amount of time to achieve a 
particular lev^l of performance. These studies Were often criticized for either, 
the lack of rigor in their experimental designs or in the descriptions of 
comparison groups which often served as controls. Many studies ernployed a 
control group which represented the "traditional" approach to teaching a 
particular subject, for example, hundreds of studies were run at that time 
which compared innovations sucH as televised instruction or programmed 
instruction with traditional/ instruction. The results of these comparisons 
were often mixed and contradictory. , 
It was in this environrnent that the federal government began to provide 
substantial funding for curriculum development projects in the late IQSO's 
and early 1%as. The intent of these projects was to bring the latest advances 
in IcnbwWge ia a variety of content areas into high school classrooms' using 
innovative curriculum development approaches. Literally millions of dollars 
were invested in these curriculum development projects prior to the 
evaluation of the new materials in the classroom. To the concern of many, 
the eventual evaluation of thesie materials indicated that^ while they were 
quite effective with very bright students, they were of limited value to 
average and oelow average students. 

This shortcoming stimulated a number of discussions among 
educational researchers concerning the procedures that might be employed 
to avoid th^ funding of implementation projects prior to determining the 
effectiveness of the innovation. The most substantial conceptual outcome of 
these concerns was tJ^ paper by Scriven (1%7) in which he discus^ the 
need for both formative and summative evaluation techniques. * 

Scriven hriaintained that for years educators had conducted summative 
evaluations to determine the relative effectiveness of completed in- 
structional products. He argued that a companion type of evaluation for- 

■■ >■ ■ . - ' ■ • 
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native evaluation, should be instituted on all instructional development 
projects. Formative evaluation refers to the gathering of data relative to the 
goals of the project throughout the life of a project Jn order that devetopers 
may evaluate products and pr6cedures and make necessary changes before 
the final version of the innovation is produced. 

Both formative and summative evaluation procedures have become 
•familiar and widely used during the past 10 years. The general distinction 
made between the two terms among evaluators is that formative evaluation 
is a process which is internal to the project and used to make revisions while 
the product is in "formative"' or development sta^e. Summative 
evaluation, however, is the deterrninatiorl of the absolute and/or relative 
effectiveness of a product. 

While thes6 definitions appear at first to be distinctly different, their use 
in practice sometimes makes them impossible to differentiate. For example, 
evaluations are sometimes made during ongoing projects to determine 
whether funding for the project vyill continue. In that sense, the decision to 
be made is a summative or ultimate one. On the other hand, some projects 
are of a continuing nature and employ continuous evaluation to revise or 
upgrade materials and therefore never seem to reach a summative stage. 
While it is sometimes difficult to make a clear distinction between these two 
processes in a 'project, it is critical to note the whole new approach to 
evaluation whicfh has been brought about because of thex:oncept of for- 
mative evaluation. \ , ' 

Given this-^acfeground on formative and summative evaluation^ It Is 
now possible t6 'place the concept of impact evaluation into perspective. 
Impact evaluation is synonymous with summative evaluation as that term is 
used by Scriven. Consider, for example, the definition of impact as it Is being 
used within the context of a teacher-training project; ^ 

WHAT IS THE POSSIBLE IMPACTOF TEACHER TRAINING? 
This is the important first question that we asked in our 
research project. The key term in the question is "impact," by 
which we mean the measurable phenomena — pf positive or 
negative value — which fallow after the completion of 
training. These phenomena would not have occurred in the 
absence of training. In the language of experimental design, 
the training program is the independent (treatment) variable, 
the impact — operational ized into an array of variables — is 
the dependent variable. (Gall, et al., 1976) 

If "impact" is considered the dependent variable in an experimental 
design, then a term is needed which describes the assessment of the im- 
plementation of the independent variable. Formative evaluation, as 
described by Scrlveh, is certainly a component of that description, but a 
broader term is required. The term "process evaluation" will be used to refer 
to those procedures and techniques which are employed while the project is 
under way in order to both describeihe Implementation of the project and 
to make changes in the project is tofmake it more effective. 

A well desigfied and executed impact evaluation is the ultimate goal o^ 
each project evaluator. However, it is the thesis of this mc^nograph that the 
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Chapter Two 



evaluation is strengthened or weakened depending upon the extent to which 
process evaluation is employed. Given this position, it is therefore worth 
reviewing the role of both evaluation and the evaluatbr within the total 
scope of an educational project. After reviewing this role, the various types 
of projects to which the techniques of process and impact evaluation apply 
will be considered. 

Relation of Evaluation to Project Implementation \ 

VVhen considering the role of evaluation activities and the evaliiator in 
an innovative project, it is worthwhile to consider the questions "Who 
evaluates?", "What is evaluated?", "When is it evaluated?", "Where is it 
evaluated?", and "Why is it evaluated?" A focus on these questions will help 
to spell out the role of evaluation and evaluation personnel within the total 
context of the project: For example, when addressing the question of "Who 
evaluates?", it may be noted that several decades ago, the investigator 
designed the study, the stimulus materials, and the dependent measure or 
test, while on current projects a separate individual, the evaluator, is em- 
ployed to perform several of these tasks. If one accepts the concept of 
process evaluation, then the evaluator should be considered an integral 
mernber of the project team with resjDonsibility to work with the entire staff 
as they formulate, develop, and evaluate th? educational innovation. The 
evaluator should not be viewed as an external, non-project persog who casts 
a value judgment upon the efforts of the' project team, but rather one who 
will work with and facilitate the project's iBfforts through the systematic 
collection of data and information which will be providedao them in order 
to improve the project. 

. The careful reader. may sense that a bias is being built into project 
evaluation when the evaluator is closely identified with the rest of the 
project ar*J considered a member of the project team. For this reason, a 
number of agencies are now requiring the use of an "external" evaluator who 
is not directly involved with the prpject to make an independent judgment 
about the effectiveness of the project. This person also answers the question 
"Who?" The distinction between the two evaluators, one internal and one 
external, is irriDprtant and wtU be^discussed at greater length In Chapter VIIL 

"What is^valuated" on' a project? The initial set of questions which 
should be raised by the evaluator relates to the planning for the project. Are 
there goal statements and do they clearly reflect intended project outcomes? 
Are the persons to be served by the project clearly identified in terms of who 
-Ihey are and where they are located? 

Another set of questions which should be raised' by the evaluator 
concerns the relationship between resources and outcomes. For example/ 
does the planned intervention appear to be adequate to bring about the 
changes as stated in the goals? Are sufficient personnel available to design 
and conduct the innovative interventl0n strategy? To what extent are in- 
struments available to assess the project goals and subgoals? 

>When the intervention is applied to a target group, the evaluator should 
determine: Is the intervention actually operational? Is the intervention being 
documented? Have needed changes been identified and implemented? Are 
changes being made based on timely and accurate data collection and 
analysis? Is data reaching project decision-makers on a timely basis? 
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the f irral type of activities carried out by the eyal^ator is in the/domain 
of Impaict evaluation, Were the objectives achi€fved?vVyhat was the/absolute 
and/or relative etfectiveness of the intervention? Wi^^at amount of 'time and 
resources was repuii'ed in- order to achieve ifhe identitfied impact? What 
, unanticipated outctjmes were achieved? ; i/'-: / , 

The next question in the who, what, whe/i, where and why ^eqii(^nce is 
"When are evaliAatlbn activities conducti^" ^^E valuation apivi ties are 
initiated at the very beginning of/ the proj«Rt and a^re corrducted con- 
tinuously until the project is completed/The task is not one /hat is simply 
cqf)c(ircted at the end of tbe project, nor is ►t planrt^ ait the beginning and 
then IrVi piemen ted at the' ehd— it is an ongoing activi^^ /• 

'"Where will the.eyaluatron take.place?" The ahsw?5e^ t|lat it will occur 
, nearly any place in Vvhich project activities are under way. The .^valuator 
should attend conferences and planning sessions; affa com/nunicate project 
*^ progress. Are staff triembers' wording orV-appropriafe ta^ks? Are products 
being com pletefl on time? The evalijafor will also found Working with the 
* target grpup im its environment. The eyajuator will obserw th^ interyention 
jrt progress and' administer the relevant; assessment insfruments, 
^^i * ' The^inal questfQn to be rai^^d Is ''Why evaluate?'^-rr why should there 
^^nJI^ etmphaifs on the role of efeluation in^tHe implementatfon of 

Ne^iji^til^ project^? Clearly ^the ftrst V^I^K^ to this questiori is the need to 
'dte^fctnMeh(: the impact which th<b pj;oj&*^ai|t had upon itV intended target 
J tt>op^lation: Evidence should be coll^c{iwJ *ef0^ changes which occur as a 
i-'rejult the W^tfTierits; mat^ri^ls, pK^rp/qe a% implemented 

during w^e projects thi^.actlvity addresses tliie acicounCability question which 
j^^ds become so impprt^nt for pir^^)jecB 

\ • neecf fbr imga^^^ V|?PeijrSftto be self evident, there 

a^:l5|^ other ^equd r^.aspnsvfo^;ertip|hasizing the role of 

evaluafejrt in a^proje^ctv!The fi^ effect which evaluation can 

have, oil overall rn|Jnagempn^^ project, The concept of process 
evaluation 'implies th^fftpiitinupu$^^^^ of data which are matle 

available to decision ni^^eiri dtfring the project. These data can be employed 
^ to determine ^he effectiytiVess of the project, project materials, procedures, 
and progress tddate. This jhfjprnj^ sllpuld suggest ways in which changes 
can be made^o ma^e project &cWvities more effective and more consistent 
with the goals which haVe^ been established. Without this type of internal 
data and enlightened decision makihg, the project director may be in the 
same position as those who directed the curriculum development projects of 
the early GCs. Often the products can appear to be well designed and well 
developed, but without feedback concerning the effectiveness of products 
from the actual users, effectiveness is still onjy hypothetical. It is the 
feedback during the formative stages of a project that indicates the extent of 
the success as w^l as the changes that must be-made to improve the project . 
The third reason fqr heavy emphasis on the role of evaluation is the 
' need to document th^. procedures employed to bring about the impact 
which is to be assessed. When one is conducting experimental research, it is 
mandatory that procedures or "t/eatments" employed with the learners be, 
clearly described so that other researchers can replicate or build on the 
conclusions of a study. It islequally important when developing an 

the developer describe and document 
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procedures employed In order to bring about the desired impact. For 
example, if the goal of a project islo develop,self-instructional materials for 
junior high students, critical material that should bie documented includes 
the procedures used to design and deyelop these materials, the data which 
indicate student performance on tryouts, revisions which have been em- ' 
pibyed based on student performance, and the effectiveness of the final 
materials. Of great concern is the description of the employment of these 
V*. • vr. materials in the classroom and any supporting materials and procedures 
* "^H^i , which, must be included to u§e them effectively. Often tfiese details are 
\ ''W* excluded from documentation reports but they aTre critical to the trans- 
\> J/.^^'^^bilitY of the materials and r 
^ % 'JIL^ example of the importance of documenting the treatment process 
%^^[°^®^ -°" ^ project can be seen in teacher training practices. Teacher 
"^W^^'^ materials, either of an in-servicf or pre-service nature, often are 
^>r^'^¥^.. aniicipation»that they ca|i be implemented in other 

^^^JJVV^acher training centers with 'similar results. The products of such training 
^r^^^Jf r projects are often only the teacher training materials and sometimes, a 
JtAtement about the effectiveness of the materials.. However, in order to 
implement the materials in local teacher centers and obtain the same degree. 
pi training effectiveness, it is quite helpful to also ha^e a description of the 
jtrainjng techniques -and the teaching philosophy^ssociated with the 
jftaining. In essence, given the material, local teacheTeducators shpuld^be 
JTOc^to approximate the teacher performance outcomes with their own 

f^f^r sumrnary, one of the greatest weaknesses in past projects involving 
^^x:at\ona\ innovation has been a lack of process evaluation data which ■ 
cbCild have helped improvethe project as it was evplving as well as facilitate / 
documentation on. the development and implementation of project ac-^'> 
tivities. Without this documentation information, we simply have the in-" 
f©rmation that an innpvatipn has been tried with a partitLil^r group 
learners, at 4 particular time, in a particular pipce, an^ that it was successi^!!j^ 
However, there is no way to replicate and use that infqrfnation within a.wide '^'^ 
range of situations because information on how to implement the in-/ 
novatibn is excluded. The task falls on the project' evaluator tp. assist in/ 
gathering Information which will make the project more effective and to 
provide datk for the potential user about product implementation anc' 
impact in the field. Given this information the educational community coulc 
employ theinnovation if there is evidence from impact evaluation that it ha 
been successful. * ■ - . 

Scope of Educational Innovation Projects / 

In this discussion of the role and scope of impact evaluation, reference 
has been made to innovate educational projects. However, there appear to 
be at least three tVpes of innovative projects with which educators are in- 
volved. These projects are those which are: (1) targeted on teacher /out- 
conr>es, (2) targeted on student outcomes, and (3) targeted on schoolwide or 
organizational.outcomes . these three types of projects share many di the 
same problems and procedures in implementing their evaluation j)\ans. 
However, each type also has its own unique characteristics. / . 
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For example,'' some goals of an innovative project which focuses on ^ 
changing teacher behavior are listed below, 

1. Upon completion of training, experienced teachers and interri^ will 
demonstrate a more positive attitude toward pupils with special 
learning/behavioral problems. 
^ 2. Teacher trainees will demonstrate an improved ability to provide for 
pupils with special learning/behavioral prohJpms as evidenced 
, through their educational plan writing and eyaluat^ by the prin- 

cipal and/or training facilitator. 
^3. One hundred (100) percent of the experienced teachers at the • 
^ ^ project site will voluntarily participate in staff " development ac- 
tivities in the training complex. Eighty-five percent of the par- 
tigpapts in training will rate the experiences as satisfactorily rrieeting 
their needs as indicated by their responses on a district-constructed 
■ * rating scale. ^ , . ' ^ 

'4. Each trainee will document the adaptation of training to his/her 
classroom atJthe end of the school year. (Smith, et al., 1976) 

These goals create very unique concerns for the evaluator who must 
determine the ^im pact of the project Quite often su<;h projects, while 
focusing on teacher training activities, have goals which relate to increases 
ijn'student performance and various other effects on the school community. 

decision to establish goals of this magnitude should not be made lightly. ' 
The University of Oregon deliberated the scope of their project and the 
extent to which they can reasonably hope to. achieve an impact beyond 
specified goals for .the teachers with whom they work. The issues rarsed are 
absolutely critical to the total evaluation process. 

;The Oregon project (Call, et al., 1976), refers to four levels of possible 
impact which a teacher training project might have. The f irstlevel of impact 
lies prirriarily in the^ number of participants who complfiite a specified 
program and their attitudes toward that program. An additional indicator of 
a level I impact would be the use of the training program by another agency. ^ 
There are a number of questions which would naturally arise when iBvel I ^ 
indicators of impact are used, such as what was the quality of the instruction 
and who were the participants. 

The questions about the nature of the program lead to the level II in- 
dicators. 6f program '^e^ffectiveness, namely teacher improvement.- teacher 
improvernent can be assessed in terms of additlorta) knowledge and skills 
and/or changes in attitude. There are many difficulties in measuring these 
changes particularly when both short-t^rm and*long-term changes are 
considered, as well as the problems of making classroom observations of 
teachers' behaviors. 

'Advocates of performance-based. teacher education have favored what 
Oregon has identified as level III indicators of project impact This level is / 
focused. on student performance and changes in behavior will occur a^ a 
result of instruction. While theofiy is not yet adequate to describe the teacher 
behaviors which will produce particular ch^pges in student behavior, in- 
creasing ,humbers of studies are being carried out which investigate these^ 
relationships. The problems wKich are present in the assessment of level *tf 
indicators are also present when making assessments at level III. 
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The fourth level of impact which a teacher training program may have is 
the influence beyond the classroom oiF an individual teacher. Because of 
changes which take place jn the classroom, other teachers may seek the 
training for thefnselves, or trained teachers niay provide special training for 
other teachers. Thes^ are obvious examples of effects beyond the classroom 
Other effects are much /nore difficult to either identify or assess^ , 

The Oregon paper clearly illustrates the difficulty which a project team 
encounters when it establishes goals beyond those of the direct populatibn. 
with which it interacts:' Similar concerns are appropriate for projects taifceted 
on students or schools. Noter^fbr example, the types of project goals Jvhich 
are listed below for student tapjget^d projects. ' 

^ Thp Murray State , Teachers Corps Project ^proposes to. ^ 
•creatie a teacher-studeni learning situation to identify and * 
maximize divergent learning poteiitials and styl^, with'the 
• goal that each child will be properly classified as 'gifted." 
(Hainsworth and Price/1976) 

ancf ' , 

■\ '. Students at Moon Middle Schoof will make an'average 
- grade equivalent gain of 0.8 during the first year of the 
project and a gain of 1.0 during the second year of the 
project as measured by scores on the Metropolitan Achieve ^ 
ment Test, Students ?t Moon Middle School will report a 
. statistical^y improved learjiing environment at the .05 level 
of confidence during #ach year of t)ie prpjedr. Students at ' 
} Moon Middle School will* sh^vv a statiitically significant 

improvement on the Oklahoma City Children'.s 6elf-Esteem 
Inventory at Jhe end of. the second year over the first year^ 
(Smith, et ^L, 1976J /- - . , / 

When a project'is focused on student outcomes^, it becomes obvious 
that the evaluation should determine whether there have been changes in 
student behavior relative to the goals of the project. When objectives 
deviate from .behavioral changes or encompass groups other than those 
directly involved, it becomes difficult to determine whether project ac- 
tivities have the^ntended effect Indirect, or inferred indices of success oft>n 
include measures such as students' attitudes, parents' attitudes, or increased 
participation by community members. So many other factors can influence 
indirect .^r 'mferred measures that these indicators provide only limited 
evidence concerning the effectiveness of the activities relateito project 
^oals. This is not to infer that goals should not be established inThese areas, 
^nor that assessments should not be made of the achievement of this type 
gcfel. However, the project director and evaluator should bfe aware of the 
lack of direct influence which Sie project tnay have upon these indicators. 

The third type of project is one which has a school-wide or system-wide 
target audience. Listed below is an examf)le of this type of project goal. 

Th^FSU/FAMU — Leon County Teacher dorps Project 
has as its research adaptation goal the, application of a 
theoretically derived vnodel for needs assessment and 
changf adoption in a public school setting (Carey, 1976), 
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This type of project presents several different, problems for the 
evaluator. These projects nriust, in fact, be evaluated' at two leveli. The first 
level of evaluation is to determine if the process which Is being employed 
has affected the recipients. The question is raised, "Have they learned the 
skills or knowledge which are to have general applicability to the education 
process?'' The "answer to this question indicates, whether the treatment 
variable was effective or had impact. "To what extent -can the recipients of 
the training utilize it to design/ impl^pient, and Evaluate subprojects yvithin 
the school?" There are many similarities between this example and the more 
specific teacher training project. The^evaluator can determine whether 
school personnel have been exposed to and have participated in' training 
activitieisi and if they. can demonstrate that they have learned the new set of 
skills. However, this is of little consequence if it cannot also be demon- ^ ^ 
strated that these hew skills in turn were used by the recipients to improve 
the instructjH^ process. Techniques must be designed to document the 
actual util jpreion qf these skills and to demonstrate that the skills were 
effective ^h learners in the Classroopri. Again, it should be highlighted, that 
there are flfficultifes in asfesing attitudes vyheh the actual recipients oif the 
"treatment" are classroom teachers, not the students, administrators* or 
community members. ' . * .« . 

Summary 

In this chapter, the changes in attitudes toward the role of evaluation/ 
and the evaluator in innovative educational projects have been portrayed. 
Evaluation has changed if rom a post facto statistical design activity to ari ^ 
ongoing process which facilitates decision-making affnong the project staff 
and uses dpcumentation tojnonitor and assess the goals of the project. It has 
been pointed out that evaluation is critical throughout the life of the project 
not only for determining the inppact of the project but aljo to plan and 
implement jDroject activities, facilitate project decision making, document ^ 
the processes which have been employed, and enthance the likelihood of 
the exportability of th^'project. 

Three different types of innovative projects have been described: those 
which focus primarily on teachers, those which focus on students, and those / 
vyhich. focus on entire organizations. Each of these projects has its own 
unique problems with regard to evaluation. However, they ail share common 
components which are to assess the first level objectives to determine the 
impact of the project, artf to deterrnine the breaHth of the effects which can 
reasonably be expected as a result of the project. ^ 
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RELATING NEEDS ASSESSMENT TO 
GOALS AND OBJECTIVES 



In the best of all possible worlds, the fi^t step in any pr.oject^flould be 
conducting a needs assessment. Needs assessment is a process which has 
become increasingly important during recent years, and is the initial process 
in devetoping criteria for evaluation. Needs assessment planriing and ac-'^ 
tivities should Involve ■ the project evaluator who should provide 
documented evidence concerning needs for the project: From these needs 
the goals, objectives, and specific activities for the project can be identified. 

Agencies which have the responsibilfty for funding innovative research 
and development projects are more and more concerned about the actual 
Reeds stated for various projects. They are beginning to require proposers to 
^ demonstrate that particular n^eds do Tn fact exist and that the projects 
proposed.wiirhelp to alleviate thosie needs. 

Numerous models have been proposed for the heeds assessment 
process. In this chap*t^r the terrpinology used In needs assessmentjBedels 
will be clarified and.one model Wilhbe presented which has been effectively 
used in the past to establish needs. The second section of the chapter will 
deal extensively with procedures which', can fee used to identify project 
needs when no formaf needs assessment has been conducted prior to writing 
the proposal. The third section will consider the procedures and problems 
involved/in converting general goals to specific project objectives. 

A Needs Assessment Process 

The word ''need" means many things to many people. While it is a word 
which is found In our everyday vocabulary, in the; Context of "needs, 
assessment" the^wbrd "need" has a -very special meaning: Most needs 
a^^ssment models require the ideritification of both the ideal status and the 
presenf status relative to an identified goal. For' example, a school distrirt 
might establish "every student reading at gradQ level" as their ideal status. A 
study of the present reading situation in the district might reveal that 40% of 
the s^derPts are reading below grade level. The gap which exists between the 
present status and the Ideal status, namely, 40% of the students do not read 
at grade level, would be identified as a need for the district. Identifying 
needs as gaps between the present and ideal status relative to a specified 
educational outcome is the mpst common use of the term need. After 
several ne^s have,beeh established; it is possible to prioritize them and to * 
id^tify specific points which can and should be addressed through specific 
directed activities such as an innovative project. * ' \ 
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^ Identified needs can be restated as goals. In our example, the fact that 
40% of the students read beloyv grade levej could be converted to a goal 
which would state that by a certain date "all student* in the district will be 
able to read at grade level." 

Experience has demonstrated that there are two elements iq the'needs 
assessment/goal setting process which appear to be critical to the success of 
innovative projects. The first element is the participation of as many dif- 
ferent clientels as ppssible in the goal setting and needs assessment. process. 
Such gr<^ps as students, teachers, administrators, parents and other com- 
munity groups can be include^. The second findlpg has been that the more 
clearly project goals are stated, the- more tarjgeted project efforts and ac- 
tivities can be toward achieving those goals, and the greater the possibility 
that success can be achieved. 

OnQ of the needs ^sessment modefs which clearTy addresses these two 
issues is the Florida Assessment and Diffusion Model (FADS). (See.Do^l, 
Kibler, Dick, Toomb, & Rollin, 1974.] The first seven steps in the FADS model 
represent a process whereby needs can be identified and prioritized. Thfe 
diagram of this' process appears in Figure. 1. 

The modelassumes that ope or more persons are working as^n agent to 
help a. client (ahytype of school group] to identify their needs. The ^process 
could be initfated by an external group such as several university professors 
vyorking with .a schookdistrict or it could be employed 1)y severc^l pen^pns 
within a school distnc't who are designated for this^ special task by the 
district. Step 1 indicates that a relationship must be established between the 
agent and the cliertt representative. Usually, the'c^i^jit is represented by the 
Assistant iSuperinteadent ^or another apfkqliijfid aaf^ That ad- 

ministrator, as shown in step 2, must identijSf?^^ th^^^ and ideal 
status of one or more problems which exist viili|itnlt^^^ In step 3, 

it must .be determined whether others withih^the^diiSnt group, such as 
teachers and other administrators, also perdeive the probt^pi as portrayed by 
the initial contact person or group, ^ ^ 
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Steps 4 and 5 are designed both to clairify and collect detailed fn- 
formation on the present status of a problem, apd to clarify the client's, 
understanding of an ideal status or solution to that problem. This step is 
critical not only because it brings clarity to the nature of the problem ad- 
dressed but also because it involves more persons in the client group than a 
single spokesman for that group. 

In st?p&^ clear description of the discrepar^cy or need which exists be- 
tween the ideal and present status is presented. In step 7 , the client should 
prioritize the needs which have been identified and Wified/and indicate 
those for which solutions will be sought.* ° 

From the use of a process ^uch as FADS, it is possible to identify goals 
which can be analyzed and prioritized by the client group. As a result, the 
client implements and evaluates an innovative project. The process is 
relatively time consuming and may result in the identification of conflicting 
views concerning the importance of various- goals withinjthe system. This 
may be particularly true when the needs assessment procedures include the 
collection of data from parents and the community, as well as teachers, 
students and administrators. However, it forces communication among 
those who will be involved and may result In a consensus decision about the 
problems wKich will be addressed. 



How to Proceed Without a^Needs Assessni]mt 

We have indicated that in the best of all possible worlds, the first step jn 
a project \s to conduct a needs assessment: The eyaluator should assist in ^ 
conducting the needs assessment process to assure an understanding of the 
goals of the project. However, in reality, time and resoarces oftQn are not ^ 
available for an extensive needs assessment Proposal deadlines often are 
too short, personnel not available, and data too. extensive to conduct a 
thorough study. Therefore, it is necessary to consider the alternatives 
available to a project when no needs assessment has been conducted. 

Perhaps, the first observation is that, if a needs assessment has not been 
formally implemented, it will . occur naturally on an informal, unplanned 
basis after the project has been implemented. - 

[t is quite conceivable that a valid need could be identified and a 
solution to remedy the need coufd be identified 'without a formal-Re^s 
assessment process. If this were the case,. then the activities addresseckto 
those needs by"lhe project staff most likely woulcl be well-received by the 
client group. As solutions to problems began to emerge, the client would 
reinforce the project staff and activities would continue according to the^ 
preconceived plan. However, if the project addresses non-existent or low 
priority needs, then problem's with project implementation are almost 
inevitable. The client who is to be affected by. the project will respcJnd in a 
neutral or negative fashion relative to their time ^jinvolvemeht and com- 
mitment to the project. They may attempt in Various ways to change the 
direction of the project to more nearly address the needs they feel are most 
critical. ' / . ' 

The following is an excerpt from a Teacher Corps project in which a 
numbej- of services were planned for school personnel without the benefit of 
an ^g>rtensive needs assessment. Note, as just ' one example, how the 
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description of the role of the research counselor changed over time as the 
real needs of the school staff became clearer, * 

■ • ^ f ' 

To illustrate iit more detail the way 'in which project 
plans have changed as the>site staW^ interacted with Rogeft 
School teachers, 'et y|^^arr|ine the -role .of the research 
, counselor. This ^ole. M^was integral to^the planning of 
two program Componej^l^lf-analysis of teaching and ysing 
conterrfporary research^ndings), has undergone major 
changes as it moved frorn planning to reality. " 

Because this was a totally new role, it was not unex- 
pected that differences ^between .the imagined role and the 
- enacted r<5le would occur. It is interesting to see the kind^ of 
adaptations that crrejbeing made as this role is integrated into 
thg^: project. 

-^..The planners conceived of this position as a way of 
: V : assisting the teachers to assimilate the new ideas tcJ* which 
. they would be exposed in this project. As originairy con- 

' : : : ceived, the research counselor would be a "r^jje^igned" 
supervisor, or a resource" per5on both trained in ^dSuhseling- 
. skills and possessing ready • knowledge of educational 
research. 

As the » project staff assembled, however, and the 
characteristics of Rogers School became better understood, 
the directors of the project altered thejr expectiatlons. Since 
one of the ultimate gpals of the prdji^ct is to have as much 
impact as, possible on* the school involved, a ^research , 
counselor was employed who was aware of the systemic as' 
well as the individual dimensions — a total human system, 
and individuals within that system. , " ' ' , 

This position tontinues to evolve. T^ie research 
CQunselor has interviewed all teachcfS from Rogers School^ . , 
who are actively involved in the project, most of the support ^ " 
personnel in the school, all of the administrative personnel,* 
and several of the teachers who do not see themselves as 
actively rnvolved in the project. He participates in planning 
activities of the project staff and focgses on the Jnterpefsonal • 
dimensions* of that interactive. process. He is ready to assist 
teasers to focus on the interpersonal dimensions of their 
inteiraction with pupils. (Morine-Dershirper, et al., 1976). 
Iri the example described, it was very fortunate that the' researcK 
cour^lor changed to more nearly meet the needs and expectations of the 
teacRh^s as the project was implemented. Other components of innovative 
projects^such as instructional materials or specif ic teaching processes are not 
nearly as flexible and, therefore, could potentially be of little or no use to 
teachers if they did not pefceive a need for the components.* ' • 

Another example of the types of problems which emerge when par- 
ticipants do not agree upon project needs, either prior to the isubmission of 
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the proposal or after the proposal has been implemented, can be presented. 
|ln the example that follows, it is clear that the university participants, the 
>sch9bl administration and the teachers all appear to have different needs 
/ and priorities which have not yet been reconciled Into a single set of 
' • targeted activities for the project. When this occurs, it is almost impossible 
Vvt to get project personnel to rtiov^ ahead on the project or to achievevthe 
cooperation of teachers in \t$ ijpplementation. 

The priorities of the school seem to differ markedly from 
those of either the Teacher Corps or the Stanford par- 
ticipants. School staff and administrators seem to be unduly 
concerned with the impending move to a new open-space 
building, originally scheduled for this spring. Although^ the 
move wiH necessitate considerable change for teachers who 
had formerly taught in self-contained classroonris, other , 
problems of great magnitude, such as ^tudents' language 
problems and low achlev^ent, seem to be neglected in 
favor ''of preparing for the move. * 

* > 

\/ The Teacher Corps' concerns with^ com'munity par- 

; * ticipation and bilingual and multi-culturafeducation are not 
\ . shared by the school staff or administration. Whereas the 
: J**" *.? !.v /^^acher Corps' seeks to develop lasting and meaningful 
.' v v <, 'nvolvemejnt^ seem threatened by parental par- g 

v.. ; / ticipatlon In school activities, and administrators appear to 
V ^' interpret "community involvement" as high attendance at 
■V^^^ ' *^^ public relations events, such as a Bicentennial 

> , V .Celebration. Granted, the school may lack personnel to carry * 
:1 ..ptJt some of the Teacher Corps goals, but;no effort seems to 
^;qfoB made to attract certificated or volunteer bilingual per- 
fortri^l (Berke, 1976). 

• 4 r • ^ . Vyheri thexlient to be affected is not involved in. the needs assessment 
process, there are constructive steps which can be Implemented to facilitate 
thfepipcess:.li).the example givenibelow, the writer discusses the integral role 
«> ^ Qf gb^ setting" and evaluation and their Importance in overall project im- 
" plementatron: In this particular project, several very broad goals which 
included changes in teachers' self-perception and behavior, were stated in 
the project proposal. However, it was necessary to focus these goals on very 
specific kinds of perceptions and behaviors. 

The evaluation of a program which has" as its outcome 
changes in teacher self-perception and behavior is very 
complex and difficult. Add 'to this the fact that those 
planning the program have nbti'ibne it before and further 
that giving the Wainwright teachers a ready-made program 
which they had no part in developing would be self-defeating 
and the enormity of the evaluation begins to becon^e clear. 
Most program evaluation specialists will insist that^efore 
any evaluation plan can be devised they must know the 
detailed objectives of the program. It shoujd be clear from 
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the foregoing discussion that the detailed objectives for the 
program were not known at the, program inception and are 
yet just dimly seen. 

In order to Ijring coherence into such an amorphous 
arena as teacher curriculum making, a number of structured 
or organizational decisions have been made which have 
evaluation consequences. The sense of powerlessness 
averred to previously springs not only from the 
.psychological, sociological, a9d political constraints placed 
on teachers and teaching. Teachers in many instances do not 
have adequate "content" knowledge to act as curriculum 
developers. Without increased knowjedge about subject 
matter and confidence in their competence in such matters, 
a program runs the risic of many "educational" programs; 
that hf^h^'ing too abstract. It was therefore decided to par- 
tition instruction into four curricular areas of concern to 
elementary school teachers . . . (Byers, 1976): 
' ■ '■ ■ - ^ - .. ^ . 

, • , !n lieu of an extensive heeds assessi^eJSit^ some proposal writers often 
employ their best professional wisdom' in^esfablishing the project goals. This 
creates a problem fq^tfie new project director in that the proposal writer, 
who may or may nW have been the pfoject director, has Enthusiastically 
written such a wide range of goals for the project that they could not con- 
ceivably be attained within a finite period of time. This poses.the problem of 
identifying only those activities to which resources should be allocated to 
bring about change. One solution to this* problem is to develop a 
questionnaire which can be administered to all affected participants in the 
^ project. The questionnaire should include all the stated project goals which 
the participants are asked to rate according to their perceived importance. 
Faced with this situation, a Florida State University evaluator'developed the 
questionnaire which appears in Table 1 (Carey, 1976). The items on the 
questionnaire were derived directly from the project proposal and were 
submitted to th|g.^htire s{aff for ranking. While there is not a great deal of 
variation in t^^e rahkinig^f,^his is one example of deriving priorities among 
project goals. A . . 

Table 1 * 

Teacher Corps Goals Assessment 

Students 

77* 1. affect attitudes toward selected -components of school program. 
85. 2. affect attitudes toward school as a whole. 
85 3. affect self-concept. 

85 4. affect achievement. • 
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Students 





c 


oiievi Deriavior. 


TL 


6. 


increase multicultural awareness. 


69 


7. 


increase intercultural experiences.^ 


92 


8>. 


improve attitudes toward schodl. 


85 


9. 


improve motivation for school — school subjects. 


77 


10, 


improve/increase aspiration level. 




11. 


improve self-concept. / 


92 


12. 


improve academic achievement — achieve a mean increase in 
achievement. * 


62 


t3. 


increase use of coping behavior. • . 




14. 


demonstrate expanded responsibllitY for self-direction in school 
related experiences. 


Teachers at Riley School^ i 


§5 


i. 


exemplify a means to improve the instructional program.* 


6? 


2: 


will demonstrate the exportability of the change system. 


92 


3. 


will be responsive to the needs &nd demands of the community. 


85 


4. 


will promote collaborative decision making between school and 
community. 


92 


5. 


will increaise beginning and experienced' teacher's capability of 
meeting the student's needs. " 


77 


6. 


will increase multicultural awareness. 


77 


7.< 


will increase intercultural experiences. 


6? 


8. 


will acquire cofnpetencies at institutional level. 


77 


9. 


will acquire competencies at the personal level. 


92 


10. 


will acquire competencies at the instructional level. 


6? 


11. 


will demonstrate increased participation in in-service programs 
through TEC. 


85 


12. 


will evidence the use of FADS and Caghe's model at Riley with 
teachers. 


6? 


13. 


will increase the ability to read, understand, evaluate, and use 
research findings. 


77 


14. 


adopt and diffuse innovations. 
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will acquire ati^ apply skills In producing educational change, such 
as individualized diagnostic/prescriptive techniques. 

become effectiiite consumers of educational research findinjgs. 

develop a critical analytic approach to such findings, 

develop and test hypotheses for educational problem solving. 

achieve a master's degree in edqcation. 

receive Fl9rida certification as arteacher. 



School Staff (Teacher Aides, Volunteers, Secretaries] 



§2 


1. 


will improve quality and effectiveness through FADS. 


92 


2. >5arry out steps to individualize instruction. 


77 


3. 


carry out steps to diagnose learner's needs and difficulties. 


46 


4. 


prescribe appropriate means of meeting identified student needs. 


§5 


5. 


implement prescribed courses of action. 


85 


6. 


develop skills in parent/community interaction, such as skills useful 
in multicultural situation. 


92 


7. 


exhibit a more positive concept of the school. 



*Perceri\ of group ranking, item "high" 



The responses from such a questionnaire can be tallied and analyzed to 
identify those target areas viewed by the participants as the most critical at 
the present time. In terms of formal needs assessment methodology, par- 
ticipants can be instructed to consider the current status of. each item, the 
ideal status of each item, and to assess the gap between the two before 
marKmg a form. For exatnple, in the FSU project the participants were asked 
to place a check mark beside the goals they considered to be most important 
to the project. An item such as "become effective consumers of educational 
research findings" was checked by 92% of the participants. It must be 
assumed that being effective consumers of educational research findings 
was judged to ,be an important ideal status and that at the present time 
participants were not considered effective consumers. * 

It can be seen from the FSU example that these types of questionnaire 
techniques require inference* and interpyetation in order to identify the 
participants' priorities among the project goals. * 

A similar, but somewhat more direct, technique for assessing needs was 
used in the Oklahoma City Public Schools to identify in-service training 
needs for teachers (Smith, et al, 1976). Several statements from the Okla- 
homa instrument are included in Table 2. Notice that teachers are requested 
to indicate the usage of each item during the preceeding year and to indicate* 
the extent they feel it will be needed during the coming year. It should be no- 
ted, however, that the items which are listed on the sample questionnaire are 



Interns 



92 


1 


92 


, 2 


§5 


3 


92 


4. 


§5 


5. 


92 


6. 
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basically sblutipns to unverified problems. For example, if a teacher should 
indicate that for the coming year there is a significant need to increase 
learning in small groups, this reflects a preference for a solution to a problenri. 
which only that teacher has identifi^. Perhaps one of the most critical 
issues .in needs assessment is to find oiit if a prot)lem is really a problem and 
not a solution to an unidentified pioblem. H sRbuld be noted in the 
Oklahoma example that, in the final palge of the questionnaire, the teachers 
are given the opportunity to respond ip. an open-ended question in which 
they are asked "to describe the real needs" of their school terms of students' 
cognitive (learning), affective (feeling), and behavioral (doing) needs; 
teachers' personal and professional needs; program needs, and so forth! This 
information would undoubtedly provide a clearer picture of the school's 
needs as seen by teachers. 



Table 2 

Oklahoma Needs Assesment Form 

« Department of Research and Statistics 

Oklahoma City Public Schools 

Moon Middle School Needs Assessment 
1975-1976 

Department Dat e 

Years Experience' " 

PARTI 

DIRECTIONS: For eachHtt^a^ of the needs assessment, you are asked to ih- 
dicate your experiences during the previous year and to assess the need 
for that item in your own school during 1975-1976; therefore, you will 
make two responses to each item. In making each paired response, 
please use the numbers for the following percentages of time which 
each item .was experienced during the previous year or is needed for the 
coming year. 




Experience With/Need for the Item 

" ~ 80-100% of the time! ' 
_ 60- 79% of the time 

40- 59% of the time . 

20^ 39% of the time 

0- 19%^of the time 
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EXAMRLE: 
Expeiience DUf^og 

a. 5 



*Textbooki in the classroom 



(Indic^Jes that 
you used texts,9 
100% of the til' 



^w^. — ' — ^- — 

DIRECTIONS: Turn to page 2 and begin 
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Need for the 
Coming Yg || f 

b.3 



(Indicates that 
, you need them 40 
. 59%ofthe\ime) 



NEEDS ASSESSMENT 



Experience 
During 
Previous 
Year 

f1974-1975) 



1. 
2. 



4., 

5.. 

6. 

7., 

8. 



17., 
18.. 

20. 
21.. 

22^ 

23., 
24.. 



i: 



Pro g ram Or g anization ^ 

S^lf-contained classes for basic skills 

Teaming within departments (Example: 
Ind: Arts projects) 

Teaming for hasjf, skills interdepart- 
mentally (Example: LA/SS) 

Skills Groups 

Self-contained for related arts 
Teaming for related subjects (electives) 
Interest groups ^ 
'Other (Explain) 



'^'tnl&^Coming 
Year. 

(1975-19761 



9.. 
10., 



Teaching Stud^nta Hnw to Learn - In^ 
dividuaiiv and in Small CnHiPt Wit^iin 
M v ClaMmom 

Learninj^^h small groups 
EstablisKihg instructional resource 
cerijters 

Developing pupil self-reliance 
Strengthelnihg teacher diagnostic skills 
Designing Individualized Prescrip- 

tibhsfor^tudents J , 

AchfeylriS joint student-teacher ; 

designs for prescriptions 
Building greater student independence 
Utilizing students' interests and 

choices for instruction 



11-. 

12. 
13. 
14. 
15.. 
16. 



• 25.. 
26.. 




30, ^ 

31. -. 
32.. 
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33 
34 
35 
36 
37 
38 



45, 
46.-. 
47.. 
48, 

'49, 

50, 
51, 
52, 
53, 



123., 

124, 

125, 

126.. 
127. 
128. 
129. 
130. 
131. 




taltlass — no grouping ' , 
l?arge groups ^ 
Small grouj^s^ 
^ . Individualizing . 
_ DepartVnental grouping 
_ Other (Explain) . ; • * 

IV. Inrtnicfmnal Tgctihjflk|g« 

Lecture^^y' , . 
^ .OemonstrSfffons and performance ' 
^ Discussion JtechniqOes , 

Intensiverinvolvement (field trif^s, 
.'class projects, etc.) . 
^ Interactive medial 
_ Instructional television 
^ Programmfed lab instruction techniques 
_ Contracts - ' , l 

^ . Community Resources 



VII. Or ganizin g to improve Studehfs Self' 

Eatecm and Indivitfual Jj^MBiiifeililyi 

_ Establishes understandingof student 
responsibilities and limits 

^ C^iposesown courses of action within 
limits 

_ Accepts failure and success with 
> positive support . 
Confidence within groups " 

Pride in school 

^ Pride in achievement 

^ Achieves at highest level possible * 

' Understands and accepts others 

Assumes various roles iri goal-oriented 
workgroups 



39., 
40. 
41: 
42, 
43, 
44, 



54.. 
55,. 
56.. 
57.. 




132., 
133, 
134, 

'137. 
138, 
139, 
140, 



'j/ PARTII 

DIRECTIONS: Considering the items in Part I and yo^ previous experiences • 
in working with the in-between-ager, describe the real needs of your 
school in terms of students' cognitive (learning), affective (feeling), and 
behavioral (doing) needs; teachers' personal and professional needs; 
program needs, etC; Finally, assign one of the following priorities-to each 
of the needs you have listed: 
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EiiflOtoQi Missirtg component or unacceptable condition that must be 
changed at the outset of the school year. ^ 

. Ciumty^ Weak corrippnent which needs strengthening during the 
coming year. ; 

Plioritv 3; Non-critical situation to be corrected over a period greater v 
than one year. 

Use the ?p ate below and the next pagrfrfo describe the needs and 
priorities of your school., ^ > 

These examples suggest that neither the project director r>or the project 
evaluator can proceed very effectively without clearly identified goals. If 
goals are not statedi in the proposal Jf they are hot stated clearly, or if there is 
a profusion of goals, then difficulties are likely to ari$e in.the project. Ex- 
perience iseems to indicate that if the participants in the project are not 
given the opportunity to help identify and clarify the goals in a variety of 
ways, they will; after the project is under way, indicate the extl^ to which 
these goa(ls are consistent with the ne^6s which they perceive in the school • 
setting. The next phase of the needs assessment process is that of converting 
goal statements into speci/ic objectives which is another activity which 
should directly involve the evaluator. , ' 



t Deriving Specific Objectives ^ 

Project objflK^es can be viewed in much the same way as Mager (1%2) 
vjewed the development of learning objectives for instruction. All the 
^ arguments which have been posed for the use of learnifjg objectives are also 
valid for the need to specify specific objectives for projects. While goal 
statements may indicate a general intent, the objectives should describe the 
specific changes which are to occur: ' 

There are basically two types of objectives which can be derived from 
general project goat statements. The first'are objectives which relate directly 
to project activities themselves. Exartiples hi this type of objective are listed 
below. • 

1. ^ Through a collaborative effort, institutions of higher education, the 

• local education agency and the community served by it will 
develop a plsm for continuous implementation of k staff 
development process at the project site and for, expansion of the 
concept to the entire school district. 

2. To establish a training complex (Staff Development Laboratory] at a 
school site through which qompetency-based .programs will be 
delivered to groups and individuals. ^ 

3. To (establish an on-going edugational personnel development 
process at the target 'site. ^ 

a. To recruit/ select, and train four interns in a specialized program • 
leading to a Master's Digree. 
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.b. To 'provide the^ educational personnel at the target site with 
training which will enable them to be more effective in dealing 
* with students of diverse ecohomicriOcial, ethnic, background 
and 'exceptionality. ^ 1 

c. To ^ovide optional graduate credit from participating in-, 
stiturions of higher education for professional development 
activities. ^ 
d; To train twenty-five community patrons for roles in service^to 
V . * the school. 

• . e. To recruit and train for^ volunteers for community service^ 
roles. (Smith, et al./1976> . 

The second type of/.objectives is directed^ specif ic'ally at t)ie target 
population to be aiffetted^y the implementation of the project. Chapter II 
includes dxamptes of specif iciy pes of objectiv^es to affect student behavior 
or performances as w<Sl as tnat of teachers. . 

The statemerft 'of spedf ic objectives is critical to any project for ^o 
reasons. Thelirst is that the objex^S^es should be used by the project director 
to focus project aftivities tow^ their achievement. For example, if one 
project objective is to make at ^st 500 self-instructional modules available 
to teachers in a teaching center,vtHien ii\Q^s must be expended to search for 
and idefitify appropriate modules and to make them available to teachers 
through the center. Likewise/if the objective is to havp all students in the 
fourth grade reading at the fourth grade level, then project fund^ must be 
expended to diagnose present reading difficulties, to prescribe instructional 
activities for identified difficulties, and provide instruction which is^needed 
to increase reading perforrnance in the fourth grade. 

The secdnd reason for the development of specific project objectives is 
that they are the cornerstone^uport which the evaluation process will be 
conducted. It is critical that the evaluator haVe such objectives available in 
order to design appropriate evaluation activities and to develop instruments . 
which will fairly and accurately measure the extent to which the project has 
been successful in achieving these objectives?. lUfi almost inevitable that, as 
the evaluator attempts to clarify goal statements through writing specific 
objectives, differences id interpretation of goals wilWemerge. The establish-^ 
ment of objectives is^n invaluable process in clarifying and specifying 
exactly what it is that the project-hopes to attain. * ^ 

From an evaluation point of view, one of the most critical aspects of 
*4!5tablishing objectives is the criterion or standard which is established for 
the objective. These standards will have a sigrilficant effect upon both the 
decision of the impact evaluation and upon the instruments which are used 
in that 'evaluation design. ' ' ' ^ 

Summary 

This chapter has emphasized the valujB.of a needs assessnnent prior to 
establishing the goals for a project. FADS was presented as»one such process 
which can be employed. A number of- after-tjierfact approaches to needs 
assessment were given with examples from various projects. After profett 
goals have been established. It is necessary to derive specific objectives in 
order to target project activities and to design the evaluation plan.. 
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DESIGNING IMPACT EVALUATION STUDIES 



Many project evaluators have had formal training in descriptive and 
inferential statistics. In these courses they^.learned the rigors of applying 
analysis of v;ariance, covariance, and multi-varjate analysis to educational 
research problerns. Evaluators are keenly aware of the need for randomized 
assignment pf . students to treatment groups and the absolute necessity of 
using control groups for any experimental study. 

The purpose of rigorous standards for the conduct of research studies 1$ 
th^t they assure, to the extent possible, the validity of the outcomes which 
are observed. However, it is almost impossible to establish the. appropriate 
research conditions to implement a true experimental design ^hin many* 
educational projects. Therefore, compromises must be made to accomrno- 
date conditions within the local setting. In this chapter, a description of the 
alternatives which are available for designing a project evaluation and a 
consideration 'of the factors which influence thie measurement of the impact 
6f the prpiect will be presented. >\ * 

When/considering whereto begin the design of the impact evaluation, it 
Is necessary to examine the project objectives as described in the previous 
chapter. The goals of almost every innovative project usuaMy include the 
developrnent of new materials or procedures and supplying^these to a client 
group. It/ Is the purpose of the evaluatpr to document the nature of the 
services provided and to assess their effectiveness and their efficiency. TKe 
doeu merj tat ipn process will be described in Chapter VI. Process evaluation 
data can ^also b^ used to determine the efficiency of services 'which have 
beeri delivered. Therefore, the assessment of the effectiveness of services 
will be thfe major concern within this chapter. 

How Objectives Influence the Evaluation Design 

Consider the three objectives listed below: 
'JA. Teachers reteiving treatment X will score significantly better on a 
^sttestthari».a group of teachers who do not receive the treatment. 

2. Teachers receiving treatment X will score, significantly better on the 
postte st than they did on the pretest. 

3. Teachers receivingireatmentX will achieve an average score of 70% 
or better on theCposttest. , 

The»wordmg of these objectives has a significant effect-^ how the 
/impact of treatment X will be determined .for. example, in, objective 1 it is 
' assujned that there will be a sufficient number -of teachers to participate 
such that half of them oan be assigned to treatment X and half bf them can 
be assigned to no trjeatmenti It is also assumed thatieach of the.teachers will 
be assigned to the two groups on a randonl basis! If teachers can be ran- 
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domly asisigned, the pretest can be administered to both groups/ the treat- 
ment can then be administered to one group, and then identical posttests ' 
can be given^to both groups,,! n this.jnstance a true experimental design has 
been established. This type of jmpact evaluation is most desirable because 
threats to Internal and external' validity of the stydy are minimized. There is 
still, the possibility that the pretest could sigtilficantiy influence the out- 
comesf 

When there Is the possibility of a preftst effect, or an interaction be- 
tween the pretest and the treatment, then It is possible to use the Solomgn 
Four-Croup design'as described by Can^pbell and Stanley {1%3). The excerpt 
below, from the Central Arkansas Teachers Corps report (Holland ahd 
Gentry, 1976), describes the application of this design to a situation in. which 
the dependient varial^le of interest Is teacher attitude, 

' ' . , f> • - : : , • 

It is probably a consensu^ that attitude caanot be 
^/ measured with the degree of precision' attainkble in > 
measuring many other human characteristics. For this and 
other reasons, the influence and impact which teacher atti- 
tude has upon the effectiveness of the instructional process 
is difficult to determine. Nevertheless, most educators con- 
. ' sider this to be a variable which exerts^ as^ubstantial and / , 
important influence upon both the aff^ctiv^ and cognitive " ^ 
developmefot of the -chrld, ^ / 

Two aspects of teacher attitude have been selected for ' 
study -;- teacher attitude toward special education students / 
r^and te^ther attitude toward curriculum. Four groups vvi 1 1 be 
utilized in the study. The groups are: ^ ^ 

^. • . . , . ^ 

• V fiuuiRj This group will consist of apprcJximately 33 
teachers erriployed at Franklin Primary School in •the Little 
. Rock Arkansas School System! This group will receive in- 
■ ; servlc.e>training in an individualized program designed to 
rmpro\;'e teacher competency in meeting the individual •/ 
needs of students in classrooms where speciar education i 
students have been mainstreamed back^into the regular 
classroom, ' ^ < , * 

filflURjl — This groupfwill consist of 'approximately 33 
teachers selected at r/ndom from^the ^Little -Rock School 
System who will be .teaching in "mainsfreamed" classrooms 
^ but will not have receivied the in-service training. 

Group III — This group will be composed of teachers ran- 
domly selected from the Little Rock School System who have , 
had no in-service training and are teaching -in classrooms . 
with no mainstreamed pupils. 

fiiaURjy — In order to compare data from teachers in a 
. different.geographical location with that of teachers in the 
Little Rock School System, Group IV will be composed of 
. teachers randomly selected from the Conway School System, 

' ■. • ■ - •' . " ' ■ ' . . 
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Croup 1 
Croup II 
Croup III 
Croup IV 



Conway, Arkansas. These teachers will receive in-service 
training similar to that provided to teachers at Franklin 
Primary School but they will not feacb in classrooms con- 
taining mainstreamed students. 

. Measurements ,wi|l be made on the above groups by 
administering both the Curriculum Affifudo Invpnfnrv and 
the Ruclcpr-namhli* F ducation Pi'p g ram Scale; Grour^ I and II 
will receive both pretest'and posttest. Croups III and IV will 
recei%ie posttest only. The design may be depicted in the 
following modification of the^Spiomon Four-Croup design: 



0. 



0. 



Where: 0 = observation (testing and measurement) 

X.^ = U.C.A. Teacher Corps in-service training 
X2 = In-service training similar to 

^2' ^3 ^ Randomly selected groups 



This design is particularly effective because the results of Croups I and 
II can be compared with those of III and IV to deterrhine the effect of the 
attitude pretest. The scores of Croups I and IV can be compared with those 
of II and II! to determine if the treatment effected teacher attitudes. The 
interaction term will indicate the relationship between the pretest' and the 
treatment. 

It should be noted that if thfere are only a small nurriber of teachers and 
if they cannot be rarjdomly assigned to the. treatnient groups, then the 
evaluator is immediately faced with the problem of attempting to create a 
valid design which will determine whether project ^obiectives have been, 

. achieved. " • v 

Now. consider objective 2: Teachers will achieve a significant gain be- 
tween the pre- and the posttest. Objectives such a$ this seem to require 
testing only of one group which would receive treatment X after receiving a 
pretest; arid prior to taking a posttest. This is an extremely weak design 
because there are numerous oth^pr factors which mifeht influence a particular 
outcome. For example?, other learning activities may be available at the same 
time the treatment is provided, anti these activities may have created the 
observed gain rather than the treatment. Even giyen this weakness, this type « 
of design is superior to one which includes only a posttest. ^ With the 
exist<pnce of a pretest, at least it can be documented that a fSarticular group 
of teachers, however chosen, could not perform the task'^efore the treat- 
ment and could perfotrti it afterwards. The major question remaining is 

''whether the treatment'itself or some other factor was responsible for the 
outcome that was observed. This matter is discussed further in Chapter VIM. 



38 



DESIGNING IMPACT EVALUATION STUDJES 



31 



In, objective 3, teachers will be assessed to 'determine if they achieved 
an average score of 70% on a posttest. This objective implies that the 
achievement of a score of 70% on a particular .test is a worthy outcome for 
teachers. The way this objective is stated, the achievement level of teachers 
prior to the treatment is unknown. This type of evaluation design does not 
necessarily require any type of statistical test. Rather, It simply calls for the 
measurement of teacher performance following sortie type of treatment to 
determine whether their performance 'at that particular time is at a stated ' 
level or standard. ' ' 

This third approach, one which sets pre-established criteria and stan- 
dards, is a relatively new approach in Impact evaluation. The West Virginia 
Teacher Corps Educational model directly addresses the ^sue of using 
criterion standards by employing measurements which are objective- 
referenced, i.e., directly derived from the project (or systenfO objectives. 
Crodsify, et al. (1976) has stated that? • 

We hope to avoid a persistent problem in measuring 
performance by not attempting to compare pupils in the. 
program with some general population of pupils. Rather, 
after the given objective^ have been determined, the / 
research is concerned with' com'paring the behavior of the 
pupils at the output with the objectives. If the behavior 
measured at the output is not what is desired or If the at- 
tendant costs are too high, dfecisions can be made ac- 
cordingly (p. 16). 

The greatest assumption in the use of criterion standards as a means of 
evaluating the effectiveness of a project is that the evaluator, project per- 
sonnel/ad^ agencies to which the project is accountable can determine a^' 
propriate criterion levels for various assessment instruments. They must 
able to identify a \^yei of performance which demonstrates to thetf 
satisfcfctlon acceptable proficiency. For some dependent measures, like a 
particular reading grade level, acceptable standards would seem j-elatively . 
easy to establish. .It is fnore difficult when the dependent measure 'is an 
attitude questionnaire for which there is no base-line or normative data. 
With questionnaires it is nearly impossible to Indicate, in advance, the level 
of response that wbXikJ be acceptable. In these cases it is almost mandatory 
to administer the instruments prior to the treatpient in order to develop base- 
Ijne data which can then be used to set a goal for the project. 

The 'Michigan State project Is an example of a projett which has 
rejected t]f\e experimental design approach to evaluation. They have instead, 
stated interest in th& achievement of certain specific skills and attitudes by 
project personnel as Illustrated in ^ description below. 

It is from the deliberation5 of tiie Evaluation Policy and 
Operations Committees that the activities of a program 
evaluation plan have^merged during the Fall of 1975. One 
very important idea to emerge frorn these activities was that 
the evaluation of the MSU-LSD 10th Cycl^ Teachers Corps 
program should not be thougJtt of as a "horse-race." Horse- . 
races always have several entrants, and the public is in- • 
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* . terested. primarily in the winner. The MSU-LSD. Teacher 
^ Coi-ps prog'rani is not being Tompared to any alternative 

program. The puestion posed was, "Can the staff of an 
elemenJtafY school learn to become effective currrculum ^ 
developers^and can they also learn the.confidence necessary ' 
to use their curricular skills?" There is nothing in this 
question which suggests a "better than" comparison. 

Since.the comparison to other programs question is.not 
implied, the evaluation must b^^ recast. to' ascertain the 
*^ change in the quality and quantity of curricular decision- 
V making by the teaching staff^ As the previous discussion 
suggests, curricular development requires a number of skills 
. .. ;*^'an(l motives. First, teachers must become aware of curricular 
... i ^ shbrt??j;^ihgs and injustices. Second, they must possesS^he* 
:-skiHs to right these wrortgs.'Third, they must believe that they 
are cohripetent. And finally, they must hav5 the skills to * 
^ effect the changes th^y want. These four^ideas provide a 

general frarn.ework from which S series of evaluation 
questions cari be devised. (Byers, 1976) ^ ^ 

Project objectives which are not necessarily /elaVed- to cognitive or 
» attitudinal outcom^^ often lend themselves to a criteifipn evaluation design 
rather than to an -€pq:jjBrimental design. As " an example of this, consider the 
objectives listed below for one'componerit df the Oklahoma Teacher Corps 
project. , ■ - 4 • * 

. * Forty (40) volunteers Vvi 1 1 b^trained for comrnunity service roles 
. whiclh ye school-related. a"' > ^ ' " 

■ / • / ''\ - -a *■ ' ' 

1. Up<*i cornpletionQf,^raihing, ea^h comfnbprty service trainee wiii 
v.oluptariJy work \vltf( school personnel in at least one on-going 
schcrbi-community activity. ^' , 

2. Upon completion Of trainir>)^, each community service trainee will 
initiate at least one school-community project involving at least 
one other community patron who did r)pt receive Teacher Corps 
training. (Smith, et al., 1976) \' ^" 

The evaluation of the attainment of these objectives consists primarily 
of the observation of forty trained volunteers in terms of their voluntary 
behavior after the training has been completed, jn essence, the indicator for 
each volunteer rs either a 1 or a Ofor each objective; they either volunteer to 
work with school personnel at least once or they do not, and they either 
initiate at least one school-community project or they do not. The design for 
this type of evaluation would simply be to determine the time, period during 
which these responses would take place and to. specify indicators which 
would serve to illustrate that volunteers had in'fact completed both types of 
activities. This data could be summarized to indicate the number of 
volunteers that achieved the objectWes and the.number who did not. While 
the data for thfcse criterion-referenced standards may be perfectly valid, the 
causes of the behaviors, are still in questioh. Can we be sure that the 
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volunteers In the Oklahoma project who achieved the objective did so 
because of the particular training they received or might there be other 
factors which would have also influenced a control group to volunteer for 
the same types of activrties? While tKis question cannot be answered, it does 
emphasize the critical need for employihg extensive project documentation 
techniques when^iterion standard designs are utilized? 

Limitations of Change Measurement 

Regardless of the approach the evaluator plans to use in an im- 
pact evaluation study, an awareness of the number of factors which will 
influence thie outcomes of that study is important. These factors have been - 
reviewed in detail in the Florida State Project report (Carey, 1976) and are 
summarized here because of their relevance to the evaluation dejsign \ask. 

The quantification and evaluation of impact or change that may be 
attributed to a specific treatment or activity is evasive when the scope of a 
study is as small as one particular treatment or a series of related treatments.^ 
However, when the evaluation of impact is attempted in large, multi-phased 
projects which involve many .groups, treatments, and expectations, the 
considerations involved are magnified, The scope of the assessment problem 
increases in fJroportion to the number Of areas in which the project is seeking 
to affect change. ' ^ 

Events' that can occur which confound the evaluation outcomes 
include the following: (1) instructional or program events that occur be- 
tween pretests and posttests, (2) non-instructional evenfs and activities 
which occur either during assessment activities or program activities, (3) the 
effects of pretesting on performance both in instructional activities and on 
the posttest, (4) the effects of statistical regression toward the mean, and (5) 
the generalizabiiity of impact within a «Decific population. 

While it is not possible to deal with each of these research design 
problems in depth (see Campbell and Stanley, 1%3, for such a discussion), it 
is important to note the more common difficulties which arise in evaluation 
research. For example, seemingly arr appropriate experimental design might 
be used in which the experimental groups receives a "treatment" which 1$ 
not received by the control group. When long term training is involved it is 
almost impossible to attribute the change which occufs between the pretest 
and the posttest solely to the treatment. Therefore careful documentation of 
project, procedures* and outcomes are required to help insure the ex- 
portability of the project results as well as the procedures. 

Other charjijps may occur during a project to effect the outcome such as 
changes in the workload of the teachers, teacher attitudes toward the in- 
struction or the project, or disruptive events in the school. These too will 
have their effect on any type of pretest-posttest assessment. 

A. third factor which may influence performance is the pretest per se. 
Feedb&ck to the teachers on their test performance or discussion about the 
test byithe teachers may result in later changes in test scores, regardless of 
the instruction. Steps should be taken to consider this interaction in the 
evaluation design, or it should be considered part of the instructional 
process, and L»ed and documented as such. - • ^ 

It is not uncommon to find a project focused on a particular sample of a 
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population which has been given a pretext. The lowest 25%, for example, 
may be singled out for special treatment. In this case, the evaluator should 
be awar^that thfere will be a statistical regriession toward t^fe mean on the 
posttest. When extreme segments of a test population, either high or low are 
^retested, even without any intervening instruction, the scores of the lowest 
students will tend to be slightly higher and the scores of the highest students 
will tend to be slightly lower, i.e., there will be a regression of the scores 
toward the mean. Therefore, the evaluator should demonstrate caution when 
interpreting the outcomes of studies which involve participants from the 
extremes In a tested population. 

As indicated in the Florida State report, and as emphasized in this 
monograph the generalizability of any research findings is dependent upon 
a complete description of the study such that an experienced educator could 
replicate it. Obviously exact people, places and dates can not be replicated. 
Therefore, the evaluator should try to ensure that there are as few ^'unique" 
factors operating in the study as possible and should describe any which do 
occur which rn ay have significantly influenced the study. 

Matching Evaluation Design to Local Conditions 

It is a major responsibility of the project evaluator to align the impact 
evaluation design with the goals and objectives which have been stated for 
the project. This should not be viewed as a one-time-on^^ask. Clearly, there 
will need to be negotiations with regard to both the design and the goal 
statements. It may be necessary to not only manipulate the evaluation 
design to fit the goals, but also to adjust project goals to the constraints for 
collecting evaluation data which exists in the project. The value judgment 
can be made that it is of grealer value to obtain valid data on a limited set of 
objectives than to have a great deal of data which have questionable validity 
for a wide range of objectives. It goes without saying that the constraints and 
limitations which are present within nearly every project'will have a serious 
effect upon the evaluation design which can be Implemented. 

It is the task of the evaluator to identify these constraints and to 
examine all possible designs which could be established to assess impact. 
The kind of design that will be uied should be established at the beginning of 
the project as well as the constraints and limitations of the design. Every 
possible precaution should then be taken during the project to minimize the 
factbrs which are likely to cause problems with the design or create possible 
extenuating circumstances relative to project outcomes. 

In selecting the evaluation design, serious consideration should be 
given to the merits of an experimental design versus the criterion standards 
approach to impact evaluation. Educators have been criticized in the past 
either for their sloppy research designs or for their impeccable designs which 
generated trivial results. Using the criterion standards approach does not 
avoid either one of these problems. The use of such standards does not avoid 
the problem of atrributing causality to changes which are observed. It does 
not prevent one from setting standards in areas which may, in fact, be trivial 
or for which no base-line data exists to judge the significance of the standard 
which has been set. However, in areas in which meaningful criterion stan- 
dards can be set and a reasonable amount of control can be established to 
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assure that the achievement of the standards is prlmarily^dftribut^ble, to the 
treatment, this approach offers a feasonable alternative t<5 othe/ ex- 
perimental design techniques. 

Unanticipated Outcomes and Goal-I^ree Evaluation 

* ■ ' .' ' . 

Many project directors and evaluators have noted that, given a two to 
three year project, the goals of the project change during the life of the 
project. This creates unique problems for the evaluator: Should the original 
Cfvaluation design be rigidly adherecl to and outcomes measured which are 
no longer relevant to the project? Should the old design be scrapped and a 
. new one established>a5 a result of the new goals for the project? 

In rnost cases, the answer is somewhere between these two extremes. 
Usually most of the original goals of the projett are nfiaintained but a new 
one is established. It is reasonable to try to develop an evaluation strategy 
for that new goal. This rjiay require the design of new evaluation instruments 
or forms. It obviously will involve some changes in project procedures and 
the treatnient implementation which are involved. These changes should be 
carefully documented so readers will understand when the change in goals 
took place, when the change in program took place, and the evaluation plan 
which was established at th^t time. 

Changes in project goals often are the result of unanticipated outcomes 
of the project which are identified through formative evaluation techniques. 
For example, it was noted in the trials of a well-known science curriculum 
that apparently because of the individualized instructional nature of the 
packages whicK were being used, the students were making very significant 
reading gains. It would seem reasonable at that point in the project to define 
a new project goal which would be '^to raise the reading level of students 
who are participating. "This new goal might or roight not result in changes in 
instructional materials but it certainly would affect the types of evaluation 
instruments that would be used and the kinds of control groups that would 
be required to assess actual gains in reading. 

Scriven (1975) has advocated that goal-fr-ee evaluatipn be used in ad- 
dition to gpal<lirected evaluation. The designs and techniques which have 
been discussed in this paper are those which are derived from g<wl 
statements and objectives and are focused tl'rectly on behaviors which are 
intended to be changed as a result of the project. Scriven argues that an 
additional type of evaluatipn should be conducted which includes ob- 
servation and assessment of the impact of the project with regard to an 
assessment of needs known to exist among that populationi Scriven has 
argued that an evaluator should be employejd who Is not knowledgabfe 
about the project goals and who uses various techniques tcudetermlne what 
is 4iappening to the target population. This method, of evaluation would 
identify many of the outcomes intended for the project as well as a number 
which may not have been intended at all but none the less are a con- 
sequence of the project. 

It is fair to say that while Scriven's ideas have merit) they have not been 
widely adopted within educational projects, The costs of'such evaluation 
Seem to outweigh the significance of the findings of such studies. Project 
directors tend to be most concerned about the degree to which their project 
has been successful in meeting the needs and goals which have been 

(1 
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identifie<r4he extent to vyhich these goals have been achieved indicates 
their success, qther outcomes which may have been realized are only of 
peripheral interest until more ejffort can be applied directly to enhancing 
these outcomes. One advantage of goal-free evaluation, however, is that it' 
may detect effects which are detrimental to the success of the pooiect and 
when discoveted, can be eliminated. 



Summary 



• Various evaluation designs hav^ been pre^sented in this chapter. When 
-possible, true experimental designs are preierred; 'however, it is oftfen 
necessary to use quasi-experimental designs. It is als<?possible to set criteria 
ot standards as the goals for the project and determine the extent to which 
these have been met. Various factors which can limit the interpretation of 
project outcomes have been discussed. The pro's and co^'sjof fiokl-ftee 
evaluation were also presented with the conclusion that this^ approaches 
otten beyond the financial capabilities of a. project. V ■ •^^ly-^. 
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CHAPTER V * 



. DESIGNING AND SELECTING * 

• EVALUATION INSTRUMENTS . 

■ ift.. " • ■ ■ 

•^ Perhaps ni> sihg^ decision made by the project evaluator is morfe 
^^itical to the: prbj^t than the design and/or selection df the impact 
ivaluattpn jnstrup^CTit^ Consider for the moment that it is not uncommon 

* lor research ^ to be funded for hundreds of 
vtf^sapds of dollars. Untold nurn6l?^of work hours, both paid and unpaid, 

are onQp^^xpended to make the project as successf uhas possible. However, it 
is not uncom[Tion to find that oirly a very small amount of time, money, or 
effort has been ajlocated to the development and/or selectiop of ap- 
propriate evaluation instruments. In essence, a description of-the success or 
failure of a verylarge project often depends upon the questionable results of 
a few poorly selected instruments. cj 

At no^point in the entire evaluation, process^is the dNigence and per- 
severance of the project evaluator more critical than at the instrument 
selection stage. The evaluator is accountably for either designing or 
selecting assessment instruments which are consistent with the goals and 
objectives of the project and which will provide reliable evidence of the 
impact, or lack of impact, of -the project. ^^N^ 

The evaluator is faced with two very basic decisions. Tnt^irst decision is 
"Which instruments willl)e used to assess project impact?", and secondly, 
"How many instruments will be used to assess project impact?" Because of 
their influence on the evaluation design, both of theSe questions must be 
answered in a rational, professionally responsible nri^;ier. 

There are basically two alternatives to the fluestjon^^^hich instruments 
should be employ^?" The first alternative is for tfei^aluator to design 
instruments yvhich match the objectives of the project. The second Is to . 
select instruments which are already developed and available. The ad- * 
vantage of developing instruments specifically for a project are that the 
evaluator can be Confident that the items included on the instrument are 
directly related to the goals of the project. The instrument can be reviewed 
by the^project staff and revised if necessary to keep it relevant to project 
goals. In addition, the instrument can be field tested with the appropriate 
target population and revised accordingly. 

The greatest difficulties in the development of new evaluation in-S. 
/strUrrient? are the problems of reliability and norming of the instrument. The 
dey^lbpnient of new instrume^itt 'fequires a great deal of effort to reliably 
and validly measure the db^tlyei whiih have been established for the- 
j)roject. In addition, there is usMi!Iy:lnsufficient time to develop and set 
norms for a test or to obtaift dii.tavt^'*. how the target population would 
typically respond to an instrument':! If a project wishes to use a criterion 
standard as part of an evaluation design, i.e., to establish a particular goal 
such as 80% dr 90% performance on a particular instrument, it is very ' 
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difficult to obtain information about the existing level of performance or 
attitudes in the target population. 

If the eyaluator is required to develop some of the assessment in- 
strumejits, it^is recommended that a subject matter specialist be heavily 
involved tn the prpcess. This helps to insure the validity bf the items and the 
acceptance of the items as true indicators of the success of project activities 
^Experience has shown that developers will often reject the results of such 
instruments as' invalid unless they themselves or another subject matter 
specialist participates i^n their development. This recommendation applies 
^ua My as well to studept attitude questionnaires 6r community surveys 
Poorly phrased questions can at best result in ambiguous data, and at worst 
lead^to the judgrT>ent that the project personnel are naive to "real world- 
conditions and therefore undeserying of careful consideration on the 
assessment form. 

In order to have an instrument which has established norms as well as 
evidence of reliability and vaKdity, the evaluator may wish to select from 
among already existing evaluation instruments. This requires a thorough 
search of standard lists of evaluatipn instruments as well as the examination 
of instruments which have been used on similar projects.- While such in- 
struments may be quite reliable Jthere are always major questions relative to 
the validity of the instruments with regard to a particular project's-^oal and 
to the correspondence between the group upon which the instrqments were 
normed and the target group in a specific project. The validity question is 
extremely critical and is perhaps most often ignored in project evaluartipns. V 

A great deal of time, effort, and money can usually be saved when the" 
eva uator can identify an already existing test which can be included^ih the. 
». evaluation design. However, it is' imperative when such a test is selected it • 
corresponds as directly as possible to the- goats of the project The conse - 
quence of selecting a test which does not focus pn the project goalsffs that • 
the time and effort spent in the project treatment are directed toward speci- 
fic outcomes while the instruments measure other outcomes. This wbuld 
obviously not be a val id evaluation of the project's direct effectiveness 

When designing and sA|cting instruments, the evaluator must deter- 
mine, "How many evaluati|| instruments will be utilized to assess the 
impact of the project?" Sometimes this question can be answered quite 
easily because there are no instruments available and therefore one-br two 
'"^truments will need to be developed especially for the' project. On the 
other hand, there ar6 sometimes many, many instruments alceady available 
or ones that can be modified to assess the project's impact 

One strategy is to use as many evaluation instruments as are available in 
order to detect any possible changes which may have occurred as a result of ' 
the project. This fs sometimes referred to as the "shotgun" approach. While 
this procedure, may result in the detection of certain changes, it Is an ex- 
tremely costly ^approach considering. the time require for respondents to 
complete the instruments and the tjjne required to record and analyze that 
•information. ' < ' .. » \: . ^ * , . - ; 

It is highly recommended/that tfW eyaluatbr b'e extremely judicious and 
choose only those instrument which are directly re'r^^^ to the outcomes of 
the project. If the instruments>eiected/pr valid and reliable, 

they should provide the inft>rm&tipn fequin^d .to evaluate the project The 
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use of a few carefully chosen instruments vyjil avoid a great de^\ of hostility 
which may arise on the part of those who are evaluated. 

Initiating, the Instrument Development/Selection Process 

The process of instrument design/selection should begin with an „ 
examination of the project objectives. Each objective should be examined 
separately and the behavior which represents the achievement of that 
objective should be carefully described. When such an analysis is complete, 
the types of behaviocvwhich emerge are usually one of the following: (a) 
cognitive outcomes, i.e,, knowledge gained by the participant as a result of 
instruction; (b) skills, i.e., the application of knowledge gained through 
instruction to particular situations; (c) attitudes towards oneself or s^ome 
, activity; (d) behavioral indicators, i.e., some action taken which indicates a 
choice of perfo^ance which is exhibited by a member of the target 
population: • ' > 

One helpful Vay of beginning the process is to draw a grid which en- 
compasses all of the goals of the project Srj^d indicates the types of 
evaluation instruments which will be needed. Below is an excerpt 
from the Michigan- State report with an illustratiOTi of such a grid. Note that 
the left column indicates the type question which should apf)ear on the 
instrument. The first category is equivalent to assiessing attitudes, vyhile.the 
second represents the testing of knowledge and cognitive skills, and the third 
* is equivalent to skill application. 



- . r * , Table 3 provides a general terTif)late for the development 

: of 'evajuation instruments,' items, and/or procedures. The 

/ ; l * • jact^^ of each curricular area's 

; : . 'evaluation in^trurnents rests with t^^ curriculum 

v; . : -. : *.deyelpp . . 

■ i ■;7 ; . : ' y the curricular development teams are to identify the 
* :^ ' V /-ceils of greatest interest by early January 1976. Following this 
, y ldentifiration,they are to construct items and/or procedures 
to assess the status of the Teacher Corps participants on each 
. ' ; . identified cell. These items from each development team will 
be combined into a single or a series of instruments and 
administered to all the participants as early in the Winter 
Term 1976 as is feasible. The safilft^ or at least parallel fqrms 
of these, instruments will be usid again in June or early 
September 1976 and finally in June 1977 at the close of the 
projects. These observations will constitute the maj6r source 
of teacher data for determining the effects of the 10th Cycle 
Teacher Corps In-service Training Program (Byers, 1976). 

^^ .■ ' ' 
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Measuring Cognitive Outcomes 

Perhaps the type of assessment most familiar to all educators is testing 
the cognitive knowledge of learners. Our techniques for measuring reliability 
and validity have been perfected for this type of assessment. Nearly all 
innovative projects which focus directly on student learning outcomes will 
use*this type, of assessment Jtechnique. Included .below is the cognitive 
testing plan which is being used in:the T^cher^Cprp? project at Classboro- 
Gamden. In this particular project, cognitive tests ar<^*adminis on a daily 
basis and norm-referenced, standardized tests are used as a pretest/posttest 
ass-essment of project impact. 

There are three frequently used ways to measure per-v 
formance. First, continuous measures\are thosevmeasures >\ 
which are taken on a dally basis. If, rapid and accurate 
program decisions have to be made, the records of the daily 
measures may be chartered and summarized as weekly 
learning. Or the-^Jaily measures may b6 stored and totaled 
e^ch week, month, or even each year. The lise of daily 
measures requires staff training. The data sheet 17601 Is us.ed 
for specific perfortnances called pinpoints /and fqr in- 
structional interventions- called phase change^. 

The second forrn of measurernent used in schools, 
triterion-referenced testing, is a measure of how many and 
what objectives are being met. The Classboro-Camden 
project has decided not to use this type of data collection 
system at present. ^ 

The th^rd type of measurement used for Ynany ongoing 
programs and projects is termed pre-post testing. Usually 
achievement tests, also called norm-referenced, are used, to ' 
measure progress. The pre-post achievement testing data to 
be . collected in this project is summarized on sheet 7603. 
(Brent, et al., 1976) 

The ClassbororCamden project typifies many projects targeted directly 
on the learner. Daily or weekly tests are especially designed for the project. 
In addition, star\d^rdized tests are used to assess the long range impact of 
the project? In the latter part of this chapter, we will return to the advantages 
and disadvantages of using standardized tests for impact evaluation. 

Measuring Skill Outcomes ■ f ■ ' ■ 

Many' projects are concerned not only with the • acquisition of 
knowledge but also the^demonstration of specific skills. These skills often 
*must be.d^monstrated in the context of a classroom. Therefore, the emphasis 
rjh'^ trairiing is. upon the application of skills in a laboratory or clinical 
r situatibrt!^' ' ' • v . 

.. / An excerpt from the evaluation plan front the West Virginia project 
(Crodsky, al., 1976] appears in Table 4. Note that the evaluator has listed 
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, / ; 'Table 4 ■ "••'^^A'^'' 

Portion of Elementary Edutation Curriculum Summafy 



cimiculuoi ^ 



student tvaluation 



1. Provide Ifcc tludcnl . 
with eipcfience in 
how cbjidrcn develop, 
learn, ^nd beluve. 

2. Provide the ttudent « 
with knowledge JMtd 
cvAluatione o< 
tc«cher4eamcr 

■ intenction, 

1 Provide the ttu- 
denl with empcriencc 
in eiecutint and 
rcpofting an ev 
pcrimcfltal ap- 
proach to tcachint> 



. lodcfMh Dl»- 
cipHnary 
Base [Part 

Ml 



1. UtUlizatlon o< 
the literaturc . ' , 
in child develop- 
nent and pty 
cholofy. 

2. lecognition and 
to ow ledge of the 



1. Um o< the IH- 



. Knowled g e of 'ihc 
major development 



mental cydct and 
the influence 
these have on 
learning and 



I AppUcaHon of 
the data in the 
Httrature to 



y iecognition and j '£v : 
knowledge of . r- 



learning theories 
and their ap- 
plication to * 
ciaMfoom practkei. 

4. Recognition and 
knowledge of modelt 
o< mcmorr aAd problemi 
o( Btorage and 
^ retrieval. 

Knowledge of the . 
' physlologk^l , 
basis of learning 
^ : and memory arid pr< 
ble ms associated 
with deficits in the 



in the classroom. 

>. Design of expert* 
mciits to be cof»< 
ducted in the' 
classroom, lab* 
oratorr> or in a 
clinical situa* 
tlon. 



dent eipeiiences 
in the analysis, 
synthesis and 
evaluation and 
application of 
research. 

2. Provide the stu< 
dent with rsper* 
iences so Hot he/ 
ihe can plan a 
kmg term instruc' 
tional program 
based upon Ihe 



Core 
(Part n 



status of the chiCl' 



1. To have knowledge of 
the sources ol lit- 
erature in education 
jni relaled «reas 
and to become *!• ■ 

. Hclent in the use 
of- these resources. 

2. To interpret dat) ^ ' 



tksj backgrounds 
and aptltodes of 
. specific pMpUi^ 

3. Provide the stu* 
- dent wifh exper- 
Jilch will 
ow for develop- 
ment of their own , 
or another model 
of iilanning and 
eaecuting leamlng 
eipertences for 
pupib based im a 
research oriented 
decisibn making 



nlng felatlva to .* 
the needs of pupUs. . 

X'lo have knowbdne of 
the application of 
different research 

1 in 
r to I 



1. Written research ' 
critique evalw«> 

tlon. ^- 

2. Exercises in^.' 
obser v ation,^ -^^ 
in lei pivuiioo, . /. ' , 
— i - *i ' 
ana prcawiloN 

of bchav&rybV 
' simulated, 

dlnlcal and 
^l^boratory e^ 

3. Conduct of fe> 
search which 



retrieve sMch data in 



4. To understand the 
application and 

" BmitatioiM of 
research 

for plan rri n g educa-^ 
tional experiences. 

5. ^1o <«i^derttand 

larittesjmong 
donal goab an 
> )ectlves.nnstruk- 
tional objectives. 



collection and 
interpretation 
of data. 

4. Evaluation of 
knowledgt on 
data gatheri)«- 



L To undetftand the 
a deciilen waking 

* in educational 
and dea»with 
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the project objectives in the left column, has indicated the course training 
goals in the center column, and in the right column has irtdicated the type 
evaluation' wjg^ch will be required, ^ ^ 

In this example from West Virginia the studffit will not be asked simply 
to proyide knowledge about research, but rather will be required to critique 
research and eventually conduct research to.jSemonstrate skills which'wer^ 
obtained through the training. ^ ^ - • 

The demonstration of a skill which has been learned is often the desired 
outcome of innovative projects. This appilies particularly to teacher training 
projects where the interest is not so much in knpwiedge about something, 
but^ rather in thVrabjjity of teachers'to apply obtained knowledge in their 
classrooms. The recent 'emphasis/placed on performance-based teacher 
education has'higWighted the imjjortance'of this type of evaluation as well 
as the difficulty of coad'uctingjJ?'Certain kinds of skills ca^ be demonstrated 
through the planning of an aalv^ty, the 'development of* a test instrument,; 
the surt^marizing of data, and so.forth. jHowever/certain teaching behaviors 
can be demonstrated only fn the classroom under normal conditions. These 
kindsxjf behaviors require observational analysis which is much moi-e costly 
and sometimes unreliable. 

The ' (^valuator is -not encouraged to shy away from observational 
analysis as a type of evaluation technique. Quite to the contrary, the attempt 
^should be made to^measure these outcohies in the classroom if that is the 
desired goal of the project. However,, the-ftvaluator should be aware of the 
diffi^^ulty in obtaining accurate, reliable data and of the time .consuming 
n^ifre of obtaining these measures. . ; j * 

Measuring Attitudes ^ 

Many projects have as a second a rv,goal the improvement of attitudes of 
the target population toward tHemselves.vtoward the schools/ the com- 
munity, or the activity in which they are participating. The Questa in- 
struments descri&^d>^elow are an example of instruments designed to assess 
change in attitudes.- ^ 

' * Quesfa I, the questionnaire for new students, is 'ad- 
" ministered at the beginning of high school. Tfois instrument 
gathers attitudinal, biographical, d^nrirQgrafphi^, and 
♦ - * • socioecQnomtc^.information and is designed to discover a -y / 
studentfs afttitudes toward himself, his peers, his previous ^ 
school, and his n'ey^' school, his hopes, fears, and aspirations, 
Questa U^- the; questionnaire for students, teachers^, and . 
administrators, is" administered to studerits and adults 
familiar with tfrel^chool. From this instrument the school 
gains information about the degree' to whicK students, 
* faculty members, and administratprs are satisfied with 
various parts of the school and with stud^t developrfrent, 
about the nature and values pf the' school's su[)groups, ^nd 
about sources of tension and dissatisfaction. By comparing; 
certain sections of both instruments, the school can measure * 
its impact upon the student's attitodes and values. QCiesta is 
designed: to assess the school, not Individuals^ so it is 



44 



V 



Chapter Five 



completed without any identification of individuals, and the 
results are tabulated and reported for groups only (Hains- 
' worth and Price, 1976} 

AA/hile the Mur^ray St'ate instrument is very general, the dklahoma 
project has employed a very targeted attitude measurement instrument 
^^^'f'Vo^^.'^'T ^^''^^^^ toJntJicatejtheir attitudes abbut themselves (Smith 
et^L, 1976). A sample of thosejtems Appear in Table 5. / 

> Tables 

OklahiDma ChiWren's Self-Esteem Inventory 

Oklahoma City Public Schools . ^ 

Name_ _ [ _School ' 

T'eacher^ ^ :Z :_P:r;iH^ • p^t^ 

Se3Cl_I Age _Race ^ ' • 



INSTRUCTIONS: ; If the statement describe? how you usually' feel/ put 'a 
check (/ m the column "LIKE ME/' If the statement jloes not describe how 
you usually feel, put a check (/.) in the column "UNLIKE ME/' There are no 
right answers. Words or phrases in parentheses add meaning to the 
statement - v * 



1. I spend a lot of. time c^aydreamiinrg. 

2. I'm pretty sure of myself. « 

3. I would, father Be myself than anyone else. 

4. I'm easy? to like. * 

5. I enjoy talking In^Jront of the^class. 

I wish I were youn^r.** ^ . 

7. There are m^hy things about myself that 
would change if I could. ' 

8. I can make up fny mind Without too much trouble. 

9. I'm a lot of fun to be with. 

10. I'm happy with (proud of) my school work. 

11. Someone usually has to tell me what to do. 

12. I can adjust to (get used to) new Aiings easily. 

13. I seldom do things that I am sorry for later. 



LIKE . UNLIKE 
ME ME 
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14. I have many friends my own age. v'* 

15. I.dotheb^t work that I can in class. 

.16.. I ddn't give in easily when I think Vrri right, r \ 

17. I can take care of myself. * ' 

18. 1^ usually happy. 

19. I would rather play with children younger than I am, 

20. I don't like to be called on in class. 

21 . I have reasons for the things that I do. 

22. ; Things are all mixed up in my life. 

• 23. I can make up my mind and stick to It. 

24. Kids like my ideas. *y 

25. I'm not doing as well in school as )'d like. 

26. ^ really like being a boy;, .{or girl). 

27. rm not ashamed of what . I am. , 

28. liike 'the way I look-; J 
'29. I like being with other people. 

>30. I s'eldom feel upset (.uneisy) In school. - 

31 . If i have something toiiayV l-say it. . c . 

32^ I don't care what happens to me. ." * ^ - 

33. I think Tm doing O.K, . . , 

34. Kids pick on me. 

35. My teacher likes me. - . 

36. j really get upSet when I'm fussed at (scolded). ' 

37v Things usually don't bother (upset) 

me for very long. ... 

38. >l can be trusted. * 

* 39.' Other people are liked better thein I am. 

40.' My school work makes me feel discouraged 
(hopeless). 




UNLIKE 

ME v:^ 



< 



^different type of instrtiment' has been developed for use in the 
Michigan State project (Byer^/l976). First^this instrument, which appears in 
Table 6, can be-used as a rie^s assessment instrument^ i.e., the teachers can 
respond to the instrument at the beginning of the project to identify areas In 
which they need additional training. The instrument can be ujed again. at the 
end of the project to de^rmine teachers' perceived increases in their own 
abilities which ^re listed oh the form. 
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, . Table 6 ' 

MSU Reading Skills Proficiency Form 

,1. Following is a I4st of skills/tasks for the teaching of reading. Given a scale 
• of 1-5 where 5 represents proficient in the skill and 1 repfesents not 
proficient, where would you place your ability to apply each task to your 
class? 



Not 
Proficient 





1 


2 


-1 


A 

. H 


c 
D 


a. selecting appropriate 

ohiprf ivPQ 












b, understanding noh- 

ln<;triirtinnal ft^ftnrc 












r. Hiaonostir <;Hlk ' ' : 


L ■ 






- , ■> ' 




d. tran<;fer hf qkilk _ 












e. develop instructional 
strategies for word 
icjentification for 

romprphptKiinn 










; - ' 


f. knowled^[e of reading 




\ 




n 




g. selection of commercijal 
rpaHpFQ 












h. modifying commercial 
reading material to 
needs of thp<;tiiHpnK 












i. get students to work 
tnHppphHpntly 












j. assess students' ^ 

progrp^ic 






J 






k. analyze specific ^ 
"rpaHln^Qkilk 












1! instruction 


> 








1 


2. Overall, how would you assess your abiiity as a reading teacher? 
Incompetent » .Co 


mpetent 


j '1 1 2 






' 1 


5 


1 



Profii 
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All of these atti:(ude forms provide what may be considered soft data. 
They represent the reported feelings and attitudes of the target population 
about their school and themselves. While these self-report* forms are often 
quite reliable and norms can be easily developed, there is always a question 
of their validity. It is recognized that participants feel a certain amount of 
social pressure to respood in a positive and constructive way 'to these in- 
struments. In addition, i^ji^ items on such instruments are often so global and 
general that it is very dlfficuk for the respondent to know epcactly to what he 
is responding and more importantly for the evaluator to know if the project 
has in fact influenced the responses. f / 

Attitude assessment inslri<inents certainly can be a valuable asset in the 
evaluation of a projects impact. They may be critical if the goals of the, 
project are directed towards attitude change. Therefore, it is extremely 
important for the. evaluator to examine each and every item on these in- 
struments to be assured thqt (1) the item is specif yb enough that responses to 
•it will directly reflect the processes employed on the project, and (2) there rs 
no reason to believe that the project itself could change a person's attiFude with 
regard to specific questions An alternative tjr supplement to attitude 
measures is the use of other behavioral indicators which are diiscussed 
below. ® ^ 

Alternative Behavioral Indicators ^ 

When attitude change is a major component of a project, the evaluator 
is encouraged to consider actual behavioral change that may be a better 
indicator of attitudinal change than simple paper and pencil responses to a 
questionnaire. For example^ if a project goal were to improve a student's 
attitudes toward school, why not look at such indirect or unobtrusive in- 
dicators as attendance, completion of assignments, or books taken out of the 
library^ rather than simply asking the student whether he likes to come to 
school? Indicators such as school attendance or a reduction in the number of 
windows which have been broken in the school are both extremely reliable 
forms of data and they are data which decisipn makers can readily un- 
derstand and aopreciate. These are indicators of the actual performance of 
the learner wit&^egard to a particular activity rather than of what may be 
simply a socially acs^ptable response on ^ questionnaire. 

A sinnilar' approicK can be used witfi community surveys which are 
intended to assess fnfgjrcit in school programs. Surveys can only ask how the 
persorf*feels about* m^ ^t'hool, but such indices as their attendance at PTA 
meetings, their participation on special school corTjpiittees, their attendance 
at athleGc events,or their purchase of .Tt^jns sold by school groups reflect the 
community's attitqrflBS about the schbbl. These items are not necessarily 
perfect indicators of attitudes, but they do represent concrete behavior by 
the respondent with re^d to the activities in question; Attitudes may be 
inferred from these peffbrrnance observations ' 



Summary 

, Tl# major purpose ol tWis chapter has been to describe a wide range of 
assessment^nstruments and to caution the evaluator to be judicious in the 
selection and/or development of ju^tho^j^j^struments which are directly 
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related to the cjbjectives of the project. Examples of instruments which were 
used to assess cognitive ancf attitudinal outcomes we^e presentee^ It was 
noted that it is often necessary to directly observe behavior changes m the 
classroom or to use. unobtrusive measures of c^ange^ in behavior. 



• CH^^PTER VI 

4!i PROCESS EVALUAT40N 

After the project evaluator has developed an Impact evaluation design 
.aind selected or* developed required instruments to be used for the design, 
the task is not over. In fact, the major role of the evaluator is only starting as 
the development of the proqess evaluation plan for the project is begun. 
Process evaluation may be considered vto be the collection of data and in- 
formation to be used to improve the quality of the project arid to document 
the major decisions made during the implementation pf the project. This 
definition implies that process evaluation is conducted from the beginning 
of the project to the end. ft mcludes the analysis of„the documentation of 
prbject goals and relevant activities, the colfection of data and Itifprmation 
from students - to innrprove the ^effectiveness of treatments, and the 
documentation of decisions which are made relative to ^he project. ' ' * 

\n thi*" chapter, examples of .methods which can be^ JH|.to co^iduct 
process evaluation both for the overall project and fV sjpltflKt student or 
teacher activities within the project will Ue co^$idei;ed.' Also, a discussion of 
the kinds of information systems which nWV be require^ to process all the 
data generated through evaluatipn ac^tiei^jilliil be included. 

■ "Ujere are ^^ntially two types 9 t tita collected in the "process 
evaluation phase. The first is a general overall type of data which provides 
information to the staff personnel and i^j^ct director concerning the 
progress the project. Such data refle^lpthe program decisions made 
during .the project and the changes, if any, due to these decisions. 

The second type of data is subordinate to the first and is one which has 
been referred to in Sm" earlier ehaptjer gis-formative evaluation 3ata. The term 
formative evaluation infers that this is a process which is used to evaluate 
specific instructional materials and activities. It is the type of evaluation 
which instructional materials, workshops, 3(nd olher special instruction 
undergo prioi' to any summative evaluation. *'The purpose of formative 
evaluation is essentially the same as that of process evaluation, whfth is to 
collect data, facilitate decision making to improve the instruction and assess 
the consequences .of those decisions duVing. the lifetime of 'the project. 
Formative evaliiatidn is clearly focused on specific, tangible products and 
^activities which are conducted in conjunction with the target population. 

Instructional Improvement . 

Any project involved in producing instructional materials should follow 
a systematic materials design process which involves testing these materials 
in their early stages with ""a small number of students to determine the ef- 
^fectiveness of the materials. This trial data is used to revise materials ifor 
• improvedeffectiveness before they are implemented with a larger number of 
students. 

Similarly, it is not unusual for. projects involved ""in teacher training 
activiti|9|i^ conduct numerous workshops. Formative evaluation data can 
collected at each training session, either through the direct assessment of 
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teachers' performance on cognitive or skill Instruments, or through more 
general evaluatloi? Instruments. These Instruments should be designed to 
focus on the specific outcomes for each workshop and to i^easure the extent 
to which the partlolpants feel that the workshop goals were achieved. An 
example of a workshop evaluation form from the Oklahoma project apoears 
in Table 7. (Smith, et al., 1976) r j kk 

Table 7 «^ / 
^ Moon Middle School WbricshcJR 

Evaluation Questionnaire 

^^"^^ ^ : Teaching Assignnrent .[ 



Highest Degree Held Year Received Years of Ex- 
perience^ 

1. Of the following sessions, check in the left hand column those who you 
feel were not appropriate»to your individual needs at this time. In the 
nght hand column, rate the degree to which you feel the session met your 



—General Sessions (Monday) * 

_Duties and Procedures 

——Classroom Management 

— -J.earning Optidns 

—Testing for Individual Qualities 

—Jndividualized Instruction/Team Teaching 

—Positive School Attitudes 

Departmental Meetings (Thurs. & Fri.) 



high low 
degree degree 



2 .^3 4 

2 3 4 

2 3 4 

2 3 4 

2^ 3 4 

2 3 4 

2 3 4 



2. Of the following sessions, check in the left hand column those in which you 
feel that group processes were employed. In the. right hand column, rate the 
^ degree to which you feel group processes were employed. 



.General Sessions (Monday) 

.Duties and Procedures 

.Classroom Management 

.Learning Options 

.Testing for Individual Qualities 

Jndividualized Instruction/Team Teaching 

.Positive School Attitudes *y^v . 

.Departmental Meetings (Thurs. & Fri.) ' 



2 
2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 

'3 
3 



4 
4 
4 
4 
4 
4 
4 
4 



prcx:ess evaluation 



high 
degree 
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low 
degree 



3. 'Of the following sessions, check in the left hand column those in vyhich 
you feel that your department goals were formulated and stated. In the 
right,hand column, rate the degree to which goal statements were ar- 
ticulated. * • . 
—General Sessions (Monday) 12 3 4 
-^Dutlesand ProceduriBS \ 1 - 2 *3 4 
—Classroom Management 1 2 3 4 
^J^earnlng Options 1 2 3 4 
««.Testingfor Individual Qualities . * ' 1 2 3 4 
..individualization of Instruction/Team , 1 2 3*4 
Teaciiing • 

-^J'ositive School Attitudes 12 3 4 



—Department Meetings (Thurs.&Ff^) .12 3 

^ 4. The yvorkshop purposes were clear to me^ ' 1 2 3 

5. Adequate time was allocated for each session. 12 3 

6. Facilitators were well prepared and enacted .1 2 3 
their roles competently. 

7. My overall impressions of the session of the workshop are: 



4 
4 
4 
4 



8. Aspects of the inservice that were most valuable to me are: 



9. Aspects of this Inservice which I feel were least valuable are: 



10. In what was was this inservice practical in terms of your being able to 
use what was accomplished this week during the school year? 



11. My suggestions for our next inservice are: 



The frequent testing of pupils can also serve as feedback in terms of the 
effectiveness of particular teacher skills which are being implemented in the 
classroom. For example, the Classboro<Iamden charts used daily for pupils 
ateo serve as daily assessments of teachers' performances on the project's 
skill objectives and facilitates their improvement in the use of these skills. 
(Brent, et al, 1976) . 

The second purpose for conducting process evaluation is to document 
major decisions and activities which take place during the project. This 
documentation is an invaluable source of information for other educators- 
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who attempt to use the products and innovations from a project. It is critical 
to know, for example, whether teachers were self-selected for various in- 
structional activities, or whether they were assigned tJ those activities by a 
principal. This procedural drfference may cause very different outcomes 
with the same materials. It is this type of information which the potential 
user must know to determine what decisions were mad^^. and the con- 
sequences of those decisions. A future user of tKe project' materials a/id 
procedures may or may not wish to make the same type of decisions, even 
though a decision to use the materials has been made. 

The use of process evaluation to provide management information to 
monitor and change a project midstream if necessary, is relatively new. 
Many project personnel will be accustomed to continuous data collection 
during the project. It is critical tbat, as early in fhe pfRject as possible, all 
personnel be aware of the need for and the usps of process evaluation data. 
If this is riot done, significant problems in communication can'occur which . 
will stifle the effects of the evaluation team. Note the following comments below 
from the Stanford project. * 



From tlWinception of our Evaluation Unit, ^e intended 
to conduct a continuous^! formative evaJuation, providing 
feedback for project improvement to various aspects of the 
project. However, we have repeatedly experienced the 
problem of not being regarded as a resource whose role is,t6 
do this. At Hoover we are regarded with suspicion. At 
Stanford, the two faculty members of the Evaluation Unit * 
also serve in the Math Work/Study Team which may be 
perceived as the more important role. The two student 
members of the evaluation unit experience difficulty at times 
in offering advice or pointing out problems to peers and 
superiors. ^ ^ 

Fear that our feedback might be regarded >as "tale^ 
bearing" often leads us to discuss problems only arnong 
ourselves, which is not having the effect of solving the 
problem. In addition, unclear lines of authority and com- 
plicated communication channels make it difficult to know ^ 
what to say to whom in order to have a particular intended 
effect, so we hold our peace. These factors have lead us to 
de-emphasize the formative aspect of the evaluation and 
concentrate on the summative both because that seems to 
be what is expected of us, and we find it difficult to over- 
come the considerable obstacles to performing an effective 
formative evaluation. (Brent, et al. 1976) 

This quote from the Stanford project emphasizes the rieed for early 
involvement of project personnel in the design of the process evaluation. It 
is critical that the rationale.for the use of process evaluation infornfation be 
clearly understood by all participants. This understanding should reduce the 
anxiety that may be caused by the constant observation and recording of 
project activities, especially those which occur in the schpols. . 
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thif exa^pl.e the weeklv.Jog.6f events indicates 4he activities and the people 
whp:ajremVoh<ed..Sbme o^^ require pvaluation and this data is 

st;or€ic|/lr>Van;ifrforrt\ati^a^^ Thjs example deafly indicates a close 

rel&tibr^hip^bfetwfeen projpct'documentat and project, management as 
indrci|tfe^^ by*Mbrm et al., 1976. , 

Tues;; ;-.:9:3C^10:0D<Bi-lingual Presentation in Dan Crowley's r6om. 
- ' V .y ■** ip^in jeljeved for interview with Greta) 

Oi "' . ■ i^^^ in Terry Clay's room. ' ■ * 

• '» v- -.(TerrY^relieyi^ 'for interview with,Mike) 

: \yed/* ^ n:,40 Assis^^ Sorensen in testing program (Eflna). 

Thjujrs;;^^^ Floyd Piper in testing program. 

' , Thurs: ' CASEV - STUDY . • ; 

&Fri. ' 1^ose,-r^, PatH^nzad - 
• ' ' ' / Karen Sorfehsen 

^.'v^ 'S^ Lupe -i- DanCrowleV 

V • / P^tHayes * 

''.V \katy— PatHanzad,^ » 

* ' Karen Sorensen * , ; 

. ' . Manuel — Floyd Piper \, 

i'Mon.: 7:00 P.M. I^arent Study Cuoup — - Ltjpe; Manuel, Rose, Edna 
. ■ ; (Jvtiguel) • • • \^^y^/ ■} ,; - 

Thuns": . 7;30 P.M. Open House. Pti^|||||^te^ 



.."^W^d. & ; Noon Recreation Program ., / . . ; :Cv' > , ^ 

Tliurs. Indoor: Dance and Movement ^ - 

J. . * Outdoor: Soccer ^' - ' ' / : v^"^ 

S-y''-.^ ' '^ '^ 

The" benefit of.ra.log is that it indicati^the type of traihihg which was 
;proyided, the amount of instruction target grbups received; aritd ^he. nijmber 
and types of special pie^tings and activities whfch were r^'quired to- im- 
plement the project. This information can be used by the p[pjek:t Hfr;ector to 
yehfy, project activrUes and to facilitate prdject decisionrttiaSingi Ih'ad- 
clitibn, it cari be shared with educators from other Jocafiortis . tip tinhanVe the 
ixpbability' of the exportability q^roject findings: . 
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In this chapter we have emphasized the importance of collecting data 
and information^yvhile the project is under way to provide feedback to the 
instructional desJgners and the project c^irectiSr, as vyell as others responsible 
for the development and assessment of various components of the projeCtT^ 
These data cart be used to facilitate project decision-making and to 
docfcypent th$ techniques and procedures employed to enhance project 
exportability. While their^-are certain problems which may arise during this 
type bf evaluation, it is ^critical thlat all project personnel understand the 
Intent of and the schedule for all process evaluation |ictivities. Otherwise, 
the lack of communication, for effective data collection or cooperation 
among personnel jpi ay cau^se importance information to be lost. 

\ ' ' ' ♦ . " - . 

Information Systems^* ; ^ 

It has been indicated that the role of the project eval^^ator includes not 
only the deslgn^nd collection of impact evaluation data, but also the data/, 
required'for process evaluation. This implies that an information system rhust. 
be available in order to smoothly and effectively store, retrieve, and analyze 
the volumes of data collected, * ' 

• Such systems should be designed as soon as possible after the initiation 
of the projett to provide a clear understanding of how the data will be collected, 
stored, and retrieved. Without such a system, the project could quite readily find 
itself in chaos— flooded with data and having no idea either why it was collected 
or. how it was to be analyzed. 

{ There are several approaches to establishing an information system. 
^ One is to adopt or modify an already existing data system to meet a project's 
.specific needs. For example, a number of projects operating in schooh 
districts^ave access to student information data bases which are owned and 
operated by the school systems.^SuCh data bases can be invaluable for 
determining academic growth, biographical information, and the prior 
success of students in the school system. To illustrate this, the Michigan 
State project, which has been working with the Lansing School District, has 
been able to use the exten«ive Lansing data for the students included in their * V 
project. 

^ This report outlines the contents of the data base that 
,has been created as part of the evaluation of the 10th Cycle 
Teacher Corps^^ Although the Lansing School District 
^ maintains records for all its elementary school children, only 
eight elementary schools were selected for inclusion in this 
data base. The criter/on for selection of the schools was that 
they had been involved in the activities of the Eighth Cycle 
Teacher Corps. Consistent with this criterion, the following 
schools were selected: Allen Street, Everett, Cier Par^f, ^ 
Gunhisonville, Holmes. Wainwright, Wexford and Willow 

The Lansing School Djstrict has been maintaining * 
achievement records on its students since 1971 A continuing 
file of data has been created by LSt) for ^ch student in the 
schools since May*of 1971. Therefore, there is longitudinal 
data available for the children c.urrentlv in the eiAt 
elementary schools composing the Teacher Corps Data Base. 
(Byers, 1976) . 
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While it is vei^ efficient to iSSSpfc usually d^%>t 

include all the information desifitfjjpiag^^ jt is usualiyi dif- 

ficult to either modify or add ctati*io ^xistrT?gJ%cords because of thelf «in- 
•ended purpose. Therefore, it is' often necessary to creMe a new inforn^iifon 
system for the project. P 

The following pages contain a check list which has been adaptedApm 
the major components of the information system which has been desSgtied* 
for the FSU project (Cajey, 1976). Note that^fe>step5 correspcmd with mal^ 
of the suggestions which have been mad^If^T' CondMCting a project'Svtotar- 
process evaluation. The chart basically] fetn^>hasiz^r the need for^ iden- 
tification of who is doing what, when, where anjd whyi Jf such a system is 
employed, then all instruments and data could be elefariy labeled arid stored- 
Appropriately. There should be rTo question as to why it was collected, how it 
would be analyzed to provide feedback to the project <taff, or how to 
describe the progress of the prc^jeci ^ . 

* Checklist for Information Management . 
System fo^ Assessment Within the FSU/FAMU - Leon County ' 
Teacher Corps Project 

1. Describe fjrojectlon rationale from proposal. 

a. Identify purpose of project ^ ^ , 

b. Identify goals and objectives ' ' ^ . . 

c. Identify needs (or projects - , • 

d. Identify activities ; . > - 

e. ^ Identify constrairits/ V : . ; ;^^..^^-v 

2. Verify purpose, needs, goals; activities; and constraints with steering 
committee. ' f.' '. . ^ ' & 

3. Describe information needs of the project. 

,a. Describe context evaluati<^n decisions which are needed 
b. Describe input evajuation decisions which are needed 
, c. Describe proq^s^ evaluation decisions which are needed 
d. Describe product ^valuation decisions which are needed 

4. Describe, assessment process. ; 

a. Identify, personnel required 

b. Describe assessment facilities, needed ? 

c. Describe materlals' and procedures for each test 

d. DescrlBe procedures for driivery tests aftd times for administration 

5. Describe scoring procedures. / • » o 

" ^ ■ . ■ •. . ." ■ " 
a -Identify scales, weights or other scoring instructions 
b. Identify scoring methods 

\^ 

6. Design data storage syits^^ 

a. Design arid document dita file^far>eete instrument 

b, Design and document data fileJdiS^ea}^: individual 
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7. Describe data analysis procedure. ^' ' 

a. Identify programs to be used to interpret each instrument 

8. Schedule assessment activities. . . - 
■» 

a. Identify exdct testing dafes, instruments and sites 

b. lde|Ui.i^j|data and time for delivery and.collection of instruments 

c. \lrte^tlfY l^jiniog time and date for personnel ^ ■ s 

d. l<lentit^ d|te f# and collection of nhstruments for scoring 

9. ' Trairr aisiessrrlent adnriinistrator^.' * * 

■ ■ ■ -* • ' , ■ ' - ■ • 

10. Conduct assessment. * \ * 

•11. , Prepare report of resglts. . ^ ' « 

a. Design display format for interim and final reports 

12. Identify all users at all levels for information available in system. 

'13. Disseminate information. . ^ 

■;;■•*!■-*• • « ■ . ^ - ■ ; . ■ 

\ . ■ ■ ' ^ 

Perhaps the most critical decision regarding information systems is 
whether such.a system should be computerized. Often there is considerable 
pressure to take advantage of the power of modern cohiputers, to establish 
the entire data base on a computer, and to have the computer analyze data 
for the project. When sufficient furjds, personnel, and equipment are 
-available, there is no question that this is an efficient and effective a|>- 
proach; however, experience has demonstrated that almbst invariably the' 
time a,nd effort required to build data records and develop programs to store 
and retrieve data are underestimated. 

Numerous books have been written about how to develop com- ' 
puteriied information systems, and no attempt will be made to present that 
information here. However, the project director and evaluator should be 
aware of the adyar^ages and disadvantages of these systems and perhaps 
should begin with iBtVery well designed description of the total information 
needs gf theptefect and the data which will be collected and stored after 

.such a system has been designed. It then is possible to identify sub- 
components of the data which niight lend themselves tQ easy com- 
puterization and analysis, and.likewise to identify those componehtSA^^teh 
can better be^ handled with a manual system. The question whi(f:|^ ^should* 
always be asked Vith 'regard to any particular data element in the system is 
"Why is this information being collected?" If a substantial answer is not 
forthcoming, then perhaps that element should be/emoved from the system. 

. Jn the long run, ft is more valuable to have' a few sets of C^al id, reliable data 
on the process and outcomes of the project than to be overwhelmed with an 
entire data bank of information which cannot be accessed or analyzed in a 
meaningful way. ' " 



^ ' * ' - (Jhapter Six. 

Summary : ' ' 

The term "process evaluation". has been used broadly in this chapter to 
describe several evaluation activities which typicafly occur prior to the 
impact evaluation. One importance step is the^fflRtiativ^ -evaluation and 
revision of the instructional materials and procedaPs^ppoUo thelr.use in the 
classroom. It was also stressed that the evafcatoMfcan enhance the 
probability that the ijnpact evaluation results will ffanft^to other locations 
by carefully documenting project activities an^ ^prtjcedures. This in-, 
formation can alsa be used by tne project directOr^^^a more effectively 
manage the project. Process evaluation activitie^jUlt i^r the generation of 
a great deal of information. Therefore, a well de^gnec^ut pot necessarily 
computerized, project information system is required, f . ' 
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REPOWING PRCJCESS AND IMPACT . 
EVALUATION OOtCOMES 

. An evaluation study is opiy ^ good as the document which repoJSts the 
results of fhat study. It is thjS^aluator's obligation to develop a thorough? 
and comiplete report both iff the actiylfies' which took-place during .the 
project and the impayirt which the project hjid on "its target population. The 
evaluator's obligation in* the report is not only to faithfujly represent th^ 
happenings of the project, buif also to meet the information need's of the 
audience or audiences to which the report is addressed. 

It is not uncommon to find an equivalent report^ which appeacs to have 
been written by one evaluator for the consumption of another evaluator. 
Often these reports contain jargon, vocabulary, and techni<;al procedures 
which tan only be understood by another highly trained evaluator. In ad- 
dition, such reports include numerous tables of data and results, but very 
little interpretation of the results. . 

Th« evaluator must be aware that most readers of evaluation reports are 
not professional evaluators. More typically they will be 'educational ad; 
mintstrators, project sponsors^ curriculum supervisors or university faculty 
members. The advice often givtfn to dissertation writers to write for ap in- 
formed educational audience which does not have expertise in their- par- 
ticular area would,seem to apply quite well to the evaluator. The evaluator 
should not attempt to write everything that happened on a project in the 
report; but rather should include those things^ whith are importafnt and 
significant to the understanding of the project and its outcomes. 

Report Formatt ^ i>r; 

Certainly there is rip common format that c^n be followed to report the 
findings of all project evaluations; however, there are a number of ^reas 
which th^ evaluator should include when reporting project outcomes. , 

The f irst area to be considered in an evaluation report is the background 
of the project. It is important to indicate the type of problem or problems 
V^hiqb resulted in the initiation of the project and to list the specific goals 
and objectives which were to be achieved by the project. * 

The second major area of a report should deal with the development 
process. This section should rely heavily on process evaluation data which 
was collected during the project. The reader should be informed of activities 
associated with the 'development of the intervention or treatment for*its 
eventual use with a particular target population. It should be indicated how 
particular rtiaterials were selected or developed and how a particular training 
program was designed, formatively tried out, and revised if necessary. 

Third, an explicit description of the evaluation design employed in the 
study should be made. It should inclf^de a complete description of the 

. . f . 59 



67 



60 



Chapter Se^en 



participants^as well as |he treatment or ihtervention prpvided to them 
Consideration %hotjld be g^^n to such questions a«: What was the nature of 
the treatment?^ Whefn vuas iradministered? Hov^ long^was the treatment? And 
who admirt^s^^rpd it? Anyspecj^l circumstapcesTAvhich' surrounded the 
implementation of the treatment also should bfe jncJtfiSed in the report. 

Fourth, project impact data should be presented. The mos^ critical 
feature of this section i^ to relate impact data directly to the stated^goals and 
.objectives of the project so the readet may determine what was to be 
achieved arid whether it was achieved. If an experimental or quasij 

^perimerktal design ^has been' employed, .the design should be clearly 
identified. Tal^lfs of means, standjard deviations, and the results of statisticaf 
test results shbuld be reported as well as a. brief interpretation of • the 
statistical test results. ' .. ^ 

If the evaluation desigrlis composed of a nuqrrber qf criterion meaisures 
such as the perci^nt^e of persons who achieved a» certain level of.per- 

^iormance,vthen;each jneasur^, tHe related objective, the .mean, and the 
standard deviation of performance with regard tp.'that measure should be 
described. . f . . \ 

.' " . ' ■ ■ 

Displaying Data ' , 

The evaluator is encouraged to use graphic fornjs of data presentation 
Whenever it is feasible to do so. It is easier for the eye^to examine a bar graph 
which displays the mean performance data from ^number of groups than it 
isjo interpffet the same data in tabular form. If p6rf9miance repeatedly has 
be?n measured over a period .of time, a graph shoi>ld be made to display the 
changes in behavior which have occurred. Try to avoiid presentirig; .repetiti\i'e 
tables-'pf data which you would not be interested in. reading or capable of 
interpreting yourself. 

Data from various attitude measures present a different type of problem 
fortheevaluajtor. Peifhaps the most meaningful approach for presenting su6n 
' data appears in Table 9 which was*, excerpted from *the Oklahoma Report. 
(Smith, et al., 1976) ^ ^ ; , ' 

In this type of table;the reader can see the exact format of the item and 
the percent of the population responding to each of the various alternatives. 
If two groups were to.be compared in terms of;their responses to such a 
questionnaire, each of the columng^^pearing on the chart in the example 
could be subdivided into Croup A and Croup ;B and the percent of each 
group's responses could be presented side by si^i^/for'comparison purposes. 

It is sometimes advisable to simply cluster responses to such 
questionnaire items into positive, neutral and negative responses and to 
deterl^ne'the number and percent of responses which fall into these^ree 
categories. ' \ ' / • 

The evaluator is faqpd with ^ different type of problem when op^g- 
ended resppnses are sojicited from participants vVith regard to a particular 
actiyity. It is advisable to ^at.^gor>ze the responses and tally the frequency of 
the &$ponse which falls in each category. Such data^^^h be presented first^in a 
sUnroanzed data table, wji^^rh displays the freque^ncy and percentage of 
reijilbrTidents indicating a parti^l^r point-of-view in their.ffee responses. Second, 
selected samples can be inserte^^b Help the reader gain d^\^T)&a\ understanding 
of the types of responses whicmwere pbtained; 
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A sample of this approach appears ift TaWe 10. (WrU, et al., 1976) 

^ ' Table 10 

Glassboro Teacher Attitude Report * 

' * Public Schools 
^ Camden, New Jersey 

< Forest Hill Elementary School 



One of the primary' values of Precision Reaching is/in * 
my opinion, the daily feedback concerning a child's progress 
or lack of it on a given math^bjectiv^. Prevldusly/ feedback - 
may hav^e l*en ol?tained on a more erratic basis — every " 
other day, once a week, elS^ Using the Precision Teaching ^ 
technique of plottingscores'on a daily basis, the teacher has 
a visual record of a child's accel^ation or deceleration. After , 
' an aim (goal) has bfeen detei;mined, itjcan be predicted with a 
large degree of accura^yjljtiv long it will take a child to reach 
the aim set for him. Theflotting of these points also helps a 
teacher to decide whether or not to move a child to a new ' 
objective, reinforce the skill being worked on, or reteach a' - 
particular objective. ^ 

The children liave enjoyed the one minute timings • 
working against time and themselves. By looking at the six 
cycle chart; a child can see how he or or she is p^gressing. 
The child has a specific ajin to strive lor and he can compare ' 
where he is in relationship to the aim set for him. 

By r^urniflg the one minute timing sheets to the ' 
children, the children can sec what errorj^.they rpade d^id 
wWch facts or problems need to be imprb^ed. 

Orifinally, the entireclass started on the same practice 
.^heet. Very quickly, the varied learning rates of the children 
became evident and groups 6fKhildren have been rrioved to 
sheets that best fit their needs. Since the te^^^ing goal is 
individual iration, the stigma of the slow learnet whose rate 
of learning is quite different from that pi the average or 
better student will be removed since jfhe child will be 
working at the level suited for him — not the class. 
■. c. ^' 

, Ellen Griffith 
Fourth Cr^de Teacher 

Discussing Evaluation Outcomes r' 

In a typical research project, the investigator summarizes research 
findings in the discussion section and then interprets those findings relative 
to the theory or problem which resulted in the initiation of the project 
Future research.^hich might be conducted as a result of these findings is 
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then discussed^The discussion section of an evaluation report should in;; 
elude many of thes€5«$ame points. For example, the evaluator should quickly 
*^summarize goals and objectives of the project and the extent to which the 
•impact data indic^e these have been acl^ieyed. Any impact which the 
itmlementation process of the particular prdject had on the results can then 
bifliliscussed. Certain changes and. decisions fnade during the project could 
be responsible for the achievement or non achievement of particular prdject 
goals., 

* It is important to note that the evaluator has the opportunity, if not the 
obligation, todiscuss observations abOutthe project results which cannot be 
substantiated by data. This type discussion should be clearly identified as 
non-data-based sets of observations but ones which seem to the evaluator to be 
contributing factors -to the project outcomes. 

The final component of the discussion could be recommendations 
^relative to future implementations of the project materials or procedures 
^These recommendations could be based upon both the process and thp 
^impact evaluation data as well as the consensus of experienced project 
personnel. The intent of this section is to provide'prospective users with the 
greatest amount of information possible to increase the probability of their 
success wi^ particular project materials and procedures. 

It is quite appropriate to include, with an evaluation report, copies of all 
the data gathering instrurnents which were used in the study as well as 
detailed data on particular mstruments which may be of interest to the 
reader. These documents should appear in the appendix of the report. 

Writing lor Different Audiences^ 

The reporf format which has been discussed in this chapter includes the 
common components found in almost any evaluation report. Particular 
agencies may require details on particular topics beyond those which have been 
discussed- above, and some components included here may be completely 
omitted from the required agency reports. However, this format represents a 
relatively general approach to the rf portiho of evaluation data. 
'■^i^ Several s^gge^tipns cai) oe made relative to additional types of reports 
whicj) may be necessary for different target audience's. For example, if an 
evaluation report hai*become lengthy, i.e., 30 pages or more, the evaluator 
may vv^nt to develop an executive summary which would appear as the first 
two pa^es of the report. An executive summary is intended to convey 
quickl^ahd clearly the nature of the problem, the goals anpl objectives of the 
proi|;ct, the treatments which have been administered, and the findings of 
^he impact of the interventions. The reader of an executive summary should 
"^fe able to understand in a general way the specifics of the project which will 
appear on succeeding pages of the reoort. 

The evaluator also should be sensitive td the need for special 
•^descriptions pf project findings for other audiences. If a report were to be 
prepared for distribution to parents whose children participated in a project, 
the genf rale valuation report might be significantly modified for this use. For 
example, the detailed description of the preparation of project activities and 
^ implementation of the protedures and programs rfiay be briefly summarized, 
while the overall problem and the irripact obtained are given rrjuch greater 



prominence in the report. In addition, technical terms should be explained in 
greater detail or avoided altogether in the report. Similar types of ad- 
justments could be made for other special groups. 

As a final comment, it should be noted that there is usuafly an inverse 
relationship between' the size of the report and the probability th'at that 
report will be read. The evaluator is thus caught in the dilemma of at- 
tempting to present^jk^xomplete and thorough presentation of the project 
and Its impact and the likelihood that a thorough, detailed report is less 
ikely to be read. Thus, the evaluator must sometimes compromise between 
these two extremes in an attempt to present the best possible rep'ort in th« 
sma lest number'of words. Whatever the compromise, the report should be 
written in a clear, lucid manner and use an interesting and informative style. 

Summary 

/ ■ 

In this chapter, the necessity for clear, precise and tajgeted reporting of 
project outcomes has been stressed. Such a report should include the 
background, goals and objectives of the project. The procedures used to 
develop project materials and procedures should be presented • as well a4 a 
clear description of the evaluation design- and the dependent variables of.: 
indicators of project success^ Care should be given to effectively disblay the ' 

51^11^"^^ "^^^l '"c«'^ ^ ^"'^"^'V and" easily be un- 

derstood by the reader. Effective discussion of the results can help to further 
clarify the meaning of the results. In preparing the project report the 
evaluator should be aware of the interests and capabilities of potenS 
readers, and prepare two or more different reports if necessary. 
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In preceeding chapters a number of techniques and procedures has 
been discussed which may be helpful to both the evaluator and project 
qJirector in formulating a^^lan for assessing the impact of a project and the, 
processes within the project resulting in thaf impact. At various points, issues 
have been touched upon that will affect the evaluation. Some of these issues 
are clear and have apparent solutions. In this chapter major issues which will 
confront both the evaluator and the pi-oject director will be presented. These 
problems are emphasized not because the solutions are readily available, but 
rather to sensitize the evaluator and project director to them. The problems 
will bexJiscussed within three general categories: problems which de<^ with 
substantive issues of the evaluation process itself; the general role of the 
evaluator within the project; and the organization of project communication 
and its' effect on evaluation activities. 

^ Substantive Evaluation Problems ' 

Perhaps the most vital issue ^faced Jby an evaluator who attempts to 
substantiate the impact of a project on a particular target population is the 
question of the generalizability of the evaluation findings. While the findings 
. might not be questioned relative to the particular time, pface, and group of 
people who were affected by the project, the (question remains whether these 
same effects can be obtained with another population. ^ 

Traditionally, experimental designs have been established to determine 
the probability that outcomes of research projects are attributable to 
chance, or whether the differences which are obtained as a result of the 
study are in fact trUe differences which exist between the group who 
received the treatment and the group who did not. However, as has been 
indicated, often it is not possible to employ a ^rue experimental design in an- 
educational project; therefore, tbe evaluator is automatically relinquishing some 
degree of generalizability of the evaluation findings. 

There are at least two major factors which influence ^he lack of 
generalizability of results. The first is the intermingling of effects of various 
components of the project. Such a situation is described in this excerpt from 
the San Jose Teacher Corps project. 

, . Our measulT|ment of the impact of unanticipated ac- 
tivities that hav(E|3pnjng up in response to requests of 
teachers and, pair«|«t5 will not be as .complete as our 
measurement of effects of tralnirtg. It would be very difficult 
to parcel out the effects of recreation programs or Special • 
interest classes on pupil learning, for example. . ^ 

ts .1.' ' 
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We can collect overall measures of changes in pupil 
attitudes toward school and toward each other by such 
simple devices as wting numbers of absences and fights on 
the playground. CKanges of this type cannot be related to a 
'specific project activity, but can indicate an impact of the 
project as a whole. (Morine-Dershimer^ et al., 1976) 
The San Jose project is a good exanr|ple of a large scale treatment which 
has many components that interact in affecting the same target population. 
Ciyen such interactive conditions, it is almost impossible to isolate the exact 
causes of differences observed on pretest/posttest type measures. Various 
types of correlation and multiple regression techniques can be applied tO'.the 
data in an effort to determine the relative contributions of the various 
factors, to the observed outcomes. However such tecniques do not identify 
causes and in some instances may even produce misleading interpretations. 

The second confounding factor is the unique nature of people who work 
on a particular project. It becomes questionable whether these persons 
could be replicated with another population and whether the outcomes are 
dependent upon the Unique characteristics of the major contributors to the 
project. Tliis is not an unreasonable question since many^projects hire 
persons with unique and special capabilities to implement 
which is related to their particular skills. If another grou|3^y^re to try to 
implement the same type of project, it would be questionabfe whether they 
would have comparable skills arKMnsights to the persons who originally 
implemented the project. 

While most evaluation studies cannot .solve these p/oblems, it is 
possible to^describe in detail the processes used to implement the in- 
tervention, theskills of personnel available, and the project outcomes. If this 
information is present, it will be increasingly possible for other ecjucators to 
replicate the study and produce similar outcomes. It would appear that the 
most substantial evidence concerning generalizability of results will come 
through actual replications of project findings. The project evaluator can 
assist in the achievernent of this goal through careful documentation of the 
instruments and procedures which are used to implement the project. 

There are a number of other issues which must a^so be faced by thfe 
evaluator. One of these is the establishment of a realistic evaluation plan. 
Often the evaluator cannot control those factors believed to be critical to 
the establishment of a true experimental design. It is clear that the eyaluator 
must bend with circumstances which are present in the project byt not 
compromise the integrity of the ^valuation. Realistic control groups must be 
established in the attempt to work with intact student populations. Various 
quasi-experimental designs are available for use in these circumstances (see 
particularly the chaper titled "Designing Summative Studies at the Local 
leveV by Peter W. Airasian in the book Evaluation and Education, edited by 
W.J. Popham, 1974). In addition, the evaluator should not overlook ^he 
possible uses of the'criterion-referenced evaluatfcfir designs in the absence of 
the feasibility of other designs. 

The evaluator must be sensitive to and prepared for a change in project 
objectives during the "tourse of the project. Occasionally feeclback is 
received by project-tJersonnel which results in the redirection of parts of the 
project in order to be more responsive to the needs of the target population. 
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While a well conceived needs assessment prior to the definition of the 
project might have prevented major changes from occurring, it still must be 
resolved if it does occur. The evaluator shpuld be prepared to modify an 
evaluation design and to redesign and implement new evaluation in- 
struments that are responsive to changing goals. 

Another issue which may face the evaluator is the assessment of long 
term gOals. Examination of many projects' goals Indicates that years and 
years of testing and observation would be reauired to adequately test 
whether they have been achieved "Since each project has ^ limited life, it is . 
usually riot possible to assess long term outcomes. Rather than ignormg « 
these outcomes, the evaluator can identify particiWar variables that predict 
future performance. An obvious example of this type naeasifrp the'iise of 
course grades in junior hl^h school to predict academic* perforyiarifce^ra 
senior high school. The project goal may be directed t&wa»tl lot)g twrn-ife 
learning. success as represented by successful graduation from hl^ school^' 'J 
However, since a project might not follow^tudents through their high school * 
years,.junior high school graces rnay be used as a relative indicator pfilioire 
'academic success. Other indtcators of a similar nature.may be identffied and 
used as predictors of future Success: • ' 

In order to carry out long term research it important to create a 
Cdirefully defined data base which contains information ovef^ nymber of 
years for, a particular set of students; The Michigaa State Treacher Corps 
project has created a data base which includes-scoresjoc^groups of students 
- called "cohorts." ' • , ' , ^ ^ 4 
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Thejdata base is composed of sd< separate cohorts of 
• elementary grade students. The first of these cohorts is ' 
cbnriposed of students who, in May of 1975, were in the sixth 
grade. When those students* graduate from high^school, the^ 
will be the class of 1981v They have fceen designated as^^' 
cohort 1981 (Cl981). Th6se children completi{|g fifth' grade.;^ 
in 1975 were deislgnated as cohort , 1982 (CJ^^^l/ .^he year 
theic dass will graduate fi-om high school, ti 
remairiinfe^cohorts were named in a similar 
cohort 1986, those^ children incthe first 
spring of 1978 the data" base Will cont 
, ^ elementary s^ool performance- on : 
' G4983, ahd C19ft4. Such data. will proL 
poftunity tp*stucft^ the growth in achfievi 
, six years bf. schooling, ,(Byers,197'fe)" 

■v ' ' ■ ■ ^ :^ ■ 

An* alternative appmpach to evaluating loi 
modify them to reflect only those things which 
the life of th|e project, It is not unteasonable^L,, 
range goafcaiaspurcjB/of (TTiptiyati(|n 
to b^.mfilsiiVed at the c6nclusi6)i,<)f the project 
ijv;rV additiopal probl^ which -may face the 

is assesBiVig outcomes baseli upo%very- snffall^ , . _ 

, population. When-ci p^ojectis focus»An'studeni outcomes, th^ population 
. of " $t;uderT^ ^itP^c^^^ is usually large erfw^H that rdsul^tan be generalized to 

• ' . . * . ' * 
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similar students. However, yyhen a special inte, 
with one student at a time or a very small 
problem becomes moredifficult. A similar prp^sje 
a teacher training program if individualized^- 
made available to teachers so they could s 
according to their interests and abilities. It wou 
would select the exact sequence of modules 
for each teacher would be different, . ^ 

Tirrief series, analysis,'j^ quasi-experimel 
by Campbell and Stanley (T963), can be usi 
ticipant in the impact study and no control g 
series design requires that several observatip 
the intervention treatment. After th^" 
measurements is taken. The design can 
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ftfegy is (employed 
^:^udents; then the 
^doe encountered In . 
|ructi(5nal materials wiere ; 
\^]y' use these materials • 
f(e})r \h^X no two teachers 
'tterefpre, the treatment 

... ■ ' ' ■ 

l^sign technique descr^^d 
hen there is only one*parr 
can be established. A time 
measures be taken pHpr to 
[triheht^ ariother series of 
iented i^s': . . ' 
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In the diagram,, each 0 represents, a? 
behavior and X represents the ihten 
This design can be applied wherf 
one teacher or one student. For exami^_ 
difficulties with a particular, student. -6 
week for four weeks to determine tl ' 
exhibited by the f^Hd. The teiacher couL« 
the use of beha^^^^odificatio[i..teefrnTi^, 
change the behavior of the ch+W/ttbserv^a* 
number of weefes tp determine . if the friequeripy 
^iieduced. ^- *' . /* 

A simTlar approach could be used to det^i 
teacher had a significant effect on a ^rmp oi 
be som0;special, materials stud/ed by th^^lT' 
In mathematics skllb, Students wduld ' 
the. training of the teacher., ther).^ 
ministered oyer a period of two 
the techniques. . r^, ^ 

In both examples describefj 
detect changes in Behavior ^ubs. 
treatj»ent. lt.is critical that a num' 
andwtewlhg the.^treatm^ent to 
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ie fTia^or purpose of the design is tp 
> the adminlstrdtion of a' papticular 
if observations are taken both ^)rior to 
^Tie , likelihood that any observed 



i.. 



ange is due to the coKpccurance of some other event. The * evaluator 
should note that causality can not be inferred from this design because there 
is nfitcontrol gr^.with whicMto compare outcomes. HoweVer, the multiple 
meh^ur^ents^Kf. beh^viojg*^ to reditfce the" possibility" of i f^lse In- 
terpretatior^ of the outcoifipr^V . v, . ^' 

The data from such a^jj^M^n 'easily be plotted on graph* paper to 
depic^t^th& pretreatment'and.posttreatment performance on the dependent 
variable. Quite^AfJefj^vthefe will beua visually significant (Effect. However 
there are certain pKtfblems in appfc}g|^5tandard statistical techniques to the 
data for analy^ft. .Campbell ^nd Sg^y (1963) address this problem and 



PROBLEMS AND issues IN IMPACT EVALUATION 



f>9 





more recent developments are described by Box anfl Jenkins (1970) ^and 
Class, Willson, and Cottman (1975), 

Certainly-other-pixjbiems than those discussed here will emerge on a 
project, but those whicibSi^ve been discussed should highlight the types of 
problems that xian ahd*do arise and some possible alternatives which are 
!:available to the evajj -'^ * • i 

•The Role of the £ 

Project staff rfitjfj^i^SpKiSually aware of the tole of a project clirector, 
content specialist,, mej9fa ''§p^^ perhaps even the evaluation 

specialist. Most perceptions of evaluators are that they are personnel who 
design or administer tests to determine whether the project is successful^ The 
role is seen as a behind-the-scenes or external role to the project. The 
• evaluator must actively seek to change these perceptions. 

Hopefully, evaluators can convince the project director, and other staff 
' personnel that they 'can be of greater service td^he project if they are 
considered as internal contributing members of the project team. One 
common reaction to such a role is that the evaluator is not knowledgable 
about the intei'vention strategies which are. being employed on the project 
and, therefore, can be of do help in their* ffrplementation. It is incumbent 
upon the evaluator to become familiar with" the intervention strategies and 
the kinds of process evaluation information vVhich will be helpful to project 
personnel. ' / 

Perhaps the most significant step which the eyaluator can take in role 
definition is to provide feedback data to the priject director and staff 
relative to project progress. It is important that these" data not imply 
judgments of worth concerning the activities but rather that it be descriptive 
of achievements and nonachievements. It is the responsibility of the project 
director and staff to use these data to make changes that seem appropriate. 
The extent to which the evaluator can contribute to the success of others on 
a project will determine, in part, the extent of his own success. 

' The External Evaluator 

. While' it has tile thesis of this nc;jonograph that tte most effective 
role for the evaluator is as an integral member of the project team, the 
question can be raised about the objectivity of such an evijluator The an- 
swer to this que'stion has been, for a number of funding agencies, the em- 
ployment of an external evaluator. This person has no affiliation with the" 
project but rather report^ directly to the funding agency, % 

The major functioa/of the external evaluator is to verify the progress of 
the project and the fir/dlngs which have been reported. This is quite' a new 
role, and one which is not yet well defined. Typicallythe exterrial evaluator 
IS interested in both the process And the impact created by the project. His 
activities closely resemble those^ of an auditor who verifies the accuracy of 
reported data. Thus the external evaluator rarely cpjlects ','new raw data" on 
a project, but rather examines existing data for accuracy. 

Tl^pre are three major advantagq^ tp a project to be reviewed by an 
external'^evaluator. The first is that the evaluator will substantiate the find- 
ings to date and verify them to th^ futiding agency. This adds obvious 



• ; _v ... 

/• :. ' * , ' ^ . ' Chapter. Eight 

credibility to the findings. Of greater Importance^ hcAver/is the^HdtiBjthood 
that the evaluator will identify problems which h» gone undetected by 
projfe^rt personnel. Through discussions with a variety of staff and users, the 
external evaluator can gain a umque:perspective on the project and iDrovide 
insight into difficulties. The third advantage of thi^ review is that often the 
external evaluator can informally suggest alternative. solutions which might 
not otherwise have been cpnsidered, ' 

. In a very direct sense/the presence of an. external evaluator produces^ 
the same concerns that, formally were felt toward the project evaluator 
however, if the project is progressing a.ccordihg to the proposed plan th^n 
the review by the externarevalu^tor can be quite beneficial 

Project Organizatim^ ^ 

It is not unusual for the orj^anization of school-based innovative 
projects to become quite comple){, Often the^e projects involve campus- 
based university professors, external .agencies, ^regional laboratories or 
> consultants, school administrator!?, teachers, and students. Such projects are 
usually conceptualized and the proposal devHope^ by a smaW numlber of 
people, while those whowvill he most affected by ..thjit project will have little 
or nothing to do with either its. planning or ftjnding. Disinterest or actual 
resentment are often the fesult of this situation! v^-' 

In order to avoid negative initial reactioi 
evaluator is urged to consider a needs assessment | 

-the project, either prior to 6r immediately after fuiTuiug, wniLd wm involve 
all the members of the aff^bted groups. This "buying in" will significantly 
affect their perception of thfe project and their cooperative partrcip'ation in 

.Comniunication 

Communication presents another area of concern which i^ documented 
here from a Teacher Corps project. 

Channels for communication about aspects of the 
^ project are unclear. The project is very complex, and un- 
dertakings must be approved at several levels Ue^re ap- 
proval is granted, fhis can lead to frustration, pailicularly 
since time for approval-granting may be very long, and 
work/study teams meet at most twice monthly. For example, 
a work/study team may wish to conduct an interest survey 
among R^rents. The'^roup may want to coordinate its survey 
with tjiat of artother group whose concerns may be similar, 
such as the bilingual education and multi-cultural education 
workAtudy teams. The Evaluation Urfit would have to be 
consulted about developing and adrnlnistering the survey, 
but the survey also would have to be approved by the Hoover 
administration and the Project Steering Committee, Ap- 
proval to spend money would have to obtained from the 
Assistant Director, and allocation of costs for d'ata analysis 
decided among the involved work/study te'ams and the 
evaluation unit. (Berke, 197^ 
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All communication.^s certainly not the responsiyJWly of the evalu4r6r/ 
There is no question that the evaluation process will be significantly affected 
by the quality of the communication within the organization. 

The problem of communication is often '^companied by the 
questioning of authority. With so many different people involved in an 
innovative project, it is not unusual for the question of who is really in 
charge to arise. Is it the project director, as indicated in the proposal? Is'it the 
school principal or another school administrator? Or is the. periodic consultant 
who comes in to help plan? Unless there is a clear indication of authority, the role 
of the evaluator will be hampered. The extent to which thpse issues "cap be 
addressed early in the project will determine the extent to which Jhey can be 



The reader could have the feeling after completing this chapter that the 
project evaluator has a ne^/ly intpossible task and/in one sense, that per- ^ 
ception is accurate. Th6 evalaa.tor wiM fa^e many difficult issues relative to 
conducting the evaluation as w^Ti as problems related to simply establishing 
a role within the project^and funcffiS||mg yviithin its organizational s^uctur^!' 
However, it is-hoped that the issues rtiised in thrs chapter will alert the ' 
project director anql evaluator to problems which have occurred in the past 
and to possible solutions 'which cap be implemented through^ cognizance of 
these issues. Plans can be made arid steps can be. taken to avoid many of 
these problems if sufficient forethought and concern are addressed to them. 



minimized as problems later on. 



Summary 




. CHAPTER IX " . 

FUTURE TRENDS IN EVALUATION 

• 'It has been, the intent of this book to suggest a number of procedures 
and techrrique's which cari'be used by a project evaluator to measure the 
Impact of. a project. In addition, the role of the project evaluator as an active 
member of the project team has been emphasized through the importance' 
of process jev^tjuatipn. If there are to be reputable and exportable results 
from projects which can fye shared w^th others, it primarily will be because 
caref§l process and impact evaluatiort^'have been carried out. ' 

It is a strong statement to say that' the replicability and exportabillty of a 
project depends heavily upon the evaluation process. However, consider the 
.tangible remains of a project after it has been completed.. These remains 
include the report of . procedures which were used in the project, any 
products that were produced or collected by the project staff, and a repgfrt of 
the results obtained using these procedures and products. i 

The project evaluator is Involved with all three of these components. * 
The evaluator. is involved in the documentation process as a team member 
who participates in Intervention plarining activities, as a member who 
determines the best possible procedures to assess and evaluate interventions 
and as the member who continually.relates intervention activities to objectives 
and outcomes. 

'teamwork*is;required for effective project documentation on a large 
project with muKiple objectives and interventions. Personnel involved in 
planning and providing/ a particular actlyity also should produce a (eport 
which includes a thorough description of the objectives, the program^ and 
the target audience. They also could provide a brief description of who 
participated and when the activity occurred. Theevaluator then can analyze 
these reports along with 'hel^gW quantitatiyie data to provide formative and 
summati\^nfprmation ori ctRH activity to staff members as agreed. That 
information could relate to: need and timeliness; performance and/or at- 
titude change in the target population; Gompatability with overall goalsgi 
compatabillty with specific objectives; or any other ^fcfmation that could 
help project members assess th^ir work on a particLilar^attiUi^^^ assess the 
progress of th^ projectin genera!-. As this docanrwn1»tfcn;:TnfqrTTiat1on*t>uild 
;^pr each activity, evidence concerriihg the growtliL.iiand effec^if eness of the 
project is created. This information will prove to b<e a valuable resourjpe for 
^information required In Jnterinxand final reports conc6rnihg overall project 
impact. ; ,^ ^\ ^ 

It is the role of the evaldator to execute the iormative eyaiuafion of ''^ 
any instructional materials which? have been developed and to provide 
authors with. Information so the materials can be improved. . * . > 

It is the role of the evaluator to design, copduct, analyze, and report the 
quantitative tesults of the inipact evaluation. As ic^ be seen from these 
descripfions, m^ny of the evaluator's -activities in process and hnpact 
evalo&tion occur at the same time or are overlapping in time. |n aciditio^, 
they require the continyal assistance and coqperation of other; teapi 
members. » • 



FUTURE TRENDS IN EVALUATION^ 73 

Clearly the ro)e of the evaluator is an imjppbrtant pne and one which, 
wheri overlooked, results in a project in which appropriate evaluation 
techniques have been totally absent or inappropriately applied. Only one 
example needs to be presented to make t>ifs point. The University of Oregon 
Teacher Corps project planned to survey existing performance-based teacher 
edupation instructional modules and to ^tlect and use in their project those 
which had been clearly validated for th^V effectiveness. At the outset, the *^ 
sta^f identified 401 available modules; however, they were able to obtain 
only 225 of thos^ identified. Even then their trpubles were not over. 

Some of theiV additional difficulties are indicated below: 

Some of the training (Packages proved very difficult to 
understand. We could not figure Out how they worked, that 
is, hpyv the various parts of the training package related to 
each other, what the sequence of training , activities was, ' 
whether a coordinator was necessary, the number of.hoUr$ 
requi[ed,etc. It would.have helped greatly; although the cost 
would be prohibitive, to invite the developers of each 
package (or their rel^resentatives) to visit our project and 
«. • demonstrate the training process. * c 

Our most difficult- problem in Phase 1 was obtaining v 
evaluation reports relating to the materials' effectiveness. • 
These "fugitive-' documents seldom are published and in- 
stead must be obtained directly fr9m the developer. Also, 
developers sometimes claim that their materials have been 
evaluated when what they mean js that the materials were 

. tried out in a small-s^ale field test in whi(fh impressionistic 
data w^re collected. It appears that relatively few n^aterials 
— of the hundreds which have been catalogued — have 
been subjected to rigorous summative evaluation. (Call, et 

/al., 1976) ' 

The Oregon project is an example of a thorough atternpt to identify 
innovative instructional materials which had undergone effective Normative 
. and saprimative evaluation. Their findings are a resounding condemnation of. 
the evaluation activities which have taken Rlac^ oh innovative projects to 
date." - , ' ^ ^. . V - 

The impossibility pf continuing to receive funds from federal agencies 
without including effective evaluation techniques for a project would seem 
to be clear. Agencies are currently under pressure to support projects vyhich 
*agree to .engage in wise planning, thorough documentation, arid, evaluation. ^ 
In essence, the agencies have an investment in the project's they fupd,^ If .. 
projects are to continue to be funded, then accountability must mcrease. 

A second and equally important reason for careful , and thproi/gh + 
evaluation of innovative projects js the Requirement of acdbuiitabllity in- 
for^irjatioa when requests are made'^fdr continued support of these projects af 
the local level. The" policy for Title III projects within the ElernerftarV>i^d 
"Secondary Education K^^ct h?s>een that at the end of three years, Ipcal 
'education agencies were required td begin to support the continuation of 
the project. This transjjidn would be most difficult without evidence con- 



cerning the growth, progress^i^d j^liffecftlveness of the project. It Is avpolitical 
reality- that if school boa/ds are to^.allocate funds for the continuation of 
innovative projects they should ^bave carefylly ^^ocumented evidence 
concerning the impact of tho^se projects before commttting scarce local 
funds. ' - • ^ \.* • ' • . 

A third and perhaps more appealing reason for engaging in extensive 
project evaluaition is the professional obligation of educators to share the 
outcomes of research and development efforts. It is clear that evaluatiorr.is , 
. an extensive process, not just in terms of hiring a person, but also,in terms of, 
the materials and supplies that are involved. In addition, extensive tirfie 
commitrnents are required of all project staff as well. as members bf«lhe. 
population who are being affected by the project. Numerous hours\3re 
required to complete testing instruments, attitude surveys, and question- 
naires, as well as analyzing the documentation of meetings, decisions, and 
other interventions in the project. - * 

Consider the situation In which a project .}ias not been evaluate and^ 
documented. Such a project would essentially mvolve a group of ediititors 
who plan and impJement a iset of activities with' a group of teachers' arid 
students. At the conclusion of the activity, the project Is finished. Cost 
analyses of^this type project have indicated that P^haps hundreds or even 
thousands of dollars were spent on the few personsiwno vyere affected by the* 
program. Without documentation and evaluatian>. there is no carry over 
benefi|;for students or teachers who did not participate in the project. Sufh* 
largefinapcial investments Irs individual teachiarV^nd students can hardly be 
justified by governrnental agencies. Therefore, project personnel have the 
obligation, upon receipt oif the project, to evaluate their efforts and to share 
tWir findings* with others. » ^ * 

It may be asked whether the role of the eyaluator will become ejasier or . 
more difficult in the future. Present trends suggest that the role" will become 
better understood and accepted. It is likely that.more and more* pi-oject 
personnel wi[l be receptive to the contributions made by evaluators and 
accept their efforts as 3n Integral part of project activities. In particular, 
process ^.valuation techniques will be better understood and accepted in 
terms pf. their critical contribution to the guidance of, the. project. Ad- 
ditionally' formative evaluation will be recognized as a critical component 
of the materials development process. , 

It is Hlcely that in the years to come educational agencies wHI benefit 
from joint efforts ^hich':^Vyt)§^ the agenciies involved. As these 

working relation^^tjlj'^'^ri^^ will be easier to organize, conduct, 

^d evaluate pr^jijcjUi^^fj^ agencies.- Tt also, is likely that 

more and ;Tlor5j|;eii/j|}M place, a number of new processes and 

impact evaluati^j|Sr^r>j^ be developed 'and im'plemiented. Many of^ 

.the unsolved pf&^^^^d IsSies which have* been discussed yi thete 
chapters; wilMn fact be resdved. 4, 
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GOALS OF THE PROJECt 

The project has as its research^'^adaptation. goal the application of a 
theoretically-derived fnodel for nefeds assessnmntafid change adopti6n. The 
model specifically 'selected for application Sfhe Florida Assessment and 
Diffusion System Mqdel (FADS).: It was^:dev^'f*d in* an R&D project at 
Florida State University by a team of profe^gJtpresenting a variety of 
disciplines. • ^ " . ^ * 

The FADS model has been modified for use in a public school setting to 
assess the information needs of .teacher educatioff decision makers and to 
increase tj}e probability that those decision makers would make appropriate 

• use of research and ^development findings and products. Success in im- 
plementing the model m a publjc sctiool setting will require.interaction an«l 
cooperation among participating members, of the Ammunlty .which 

• includes the Flor^da^State Department of Education, FSU, FAMU, the Leon 
County Teacher Education Center, and Leon County school personnel. In-, 
teraction among p/dfessional educators* at theife varying JeWls should m- 
aease the variety of resources available to eac|j''''institution and imprdve 

* general educational practices in the cpnjmujpi'ty at all levels. ^ 
In addition to th^a'jQx go^\ of implementing and monitoring the use of 
the FADSmodel, there are several other goals for the^project. They include:^ 

1. Facilitating the task of identifyinjg priority Jpacher trai||.ing and { 

retraining, and delivering nece^ary solbtiorxs. it- 
2^ Designing aftd demonstrating a resear'ch-based alternf|tive form of 
« r teacher education. .5* ^1'' 

3. Jmprovin* classroom in^jtructi^n in, local Rt^blie schools and teacher 
education^ programs. ^ ^. ' "^'^ . ^ 1%;. 

4. Producing positH'e changes \in commyn% attitudes toward the 
• school and increasing ^femrrmnity p&ticipation in school-related 

^ activitiejii , • , 

" T^EATMENTINTERVENTIOf^ 

Treatments irdtlude'both formal instru.ction and applied team work in 
identified task forces. Si^ff-cehter'ed and campus-based instruction make up 
the instrii?ftional component. Faulty members from both FSU and FAMU 
.^wor|c with teachefc and Iteacher Corps interns in various workshops and 
other instructional activities identified as desirable through thte needs 
asse$sj>>ent activities of'Kiley School teachers and administrators, faculty 
^f.^from FSU arid FAMU, pirsonnel from, the Leon County Teacher Education 
Center and support personnel from teacher Corps. 

^ore ^ecifically, instructional programs through the summer' and ' 
schwl year include the following areas: (a) community/social action; (b) 
inte^rsontl skills; (c) research cohsunrTption; (d) interpreting and using 
results of diagnostic tests for lesson planning;,, (e) instructional design, 
di'ignosis, and prescription; (f) needs analysis techniques for the clas^rdorn 
atid school; (g) generic classroom teaching skills; (h) knowledge and skills in 
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^various instructional models (Bruner, Taba, Rogers, etc ); (i) curriculum 
rel$ited skiUs (math, ^cience, etc.); and (j) communication techniques with 
studeijts, parents/p^rs, admfhistrators, and cojnrriunity members. ' 



TARGET POPULATION 



There are several target populations t6 be affecteJ||tr6ugh the projects 
The overall goal36f interaction and integration of var^S members of the, ^ 
Leon County eduCatiop community has alfes targetvpopi^^lbaj Florida 
State I)epartment of Education, the Colleges of !'Eddcatio«j^t'i®rida State 
University and ..Florida Agricultural and Mechanical O'niv*^: 
County School System, a'nrfthe Leon County Teacher Educatio^'^ 

Riley Efementary SchooJ is the school site iqr which the a3j^ 
model is being applied. Interface, among the various^con^pon^ 
educ^ion'ct)mmuhityinTallahassee takes plafceatthe^khpolsite.;. 

Specific target audienc^^s for specif ic goals incljud)?: (a) students a! 
School; (b) classroom teachers at, lW|^5yopl; (c) Teacher Corps iqit 

Riley School; ((i) Teacher Corps sta""' "-^J- ^-u^^i _ 

(f ) the instructional faculty and adi 



of FSU and FAMU/:5^d (g) the Leon Cq» 

V . ^ SUCCESS 1n[ 

Specific success indicators have bee,A"L 
^.target population. The types of indicatorll 
■summarized here. ' m 



ijiey.Soh obi community mei 
jrs.of the Colleges'of Edi 
acher Education Centepj 




j|fb r eic h ^al Ari^ 
wtth eefth gi'pfip ar'i 



^udents ' \ ■ ■ 

.Success indicatoxs which ha%^b.^e§'\i^r#^^ t'd. date.ftc^'uW f©r' 



Ssessing Riley School students' -growjb: <:i%ij|3i8^lmance^ .ach(*v^rp 
* score?^, indices of self-concepts-indice^ of ihtW^iitV, a^Mienclance; ccwunselor's 
. observations, and teacher's ol^ervations of ' stddShti? b^^havjor; - . 
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e|;tiyes, status; ^bnrffhuni 
i^ncfe ' jecoms, at 
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Si 
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specified 
specified' 

skills, intern interests, grojl^ct y 
patterns, teaching proOTj^re: 
initiated ahd carried f^%i*gh 

"I :i&.m-'^i . . . , , 

-jSchobl staft;^in,clgct^i; cla^room teachefs; the ciil 
^o/dinato%^achers' aides,, and , schbol adnr>inisi 
^ess for 
r^fmservice 'activities 

- „ — :grbups; observations of classroom t 

formance^fpbservatiorts of facilitating-^jhymav^ f6f schob 



interaction 
'activitfes 



^ Riley Schf 

The; 
curricttli 
dicators 
'tendance'^3C 
assigned 5a^,k 



?lor and- 
tprs, 1n- 



lervice activities wit^h thjs group • i^iclude:. at«^^ 
/ities; Ideas initiated ^VilF carried '.through as 
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/ l^rojects; voluntary attendanc^af school* activities; partidpatidp -in "Task 
V Group meetings; participation'ih Gou^Jnseryicip.^dtivities; par^icipatioh in 
• -courses at FSU and FAMU; and per(^.ipnj: ^escribed thrlfefugh" iriventqrjes 
such as interest, motivation, teaching skinsjNclasSroom^ha^ 
teraction patterns, . , ,. • ' W " 




Teacher Cpr^^ Staff 

The TeacheWcorps staff incTudj^': 'bie 'director valuator,, researph • 
specialist, instructfongj faculty from €5U' aipd FA>^U; tf>e site cpordijnfator/ . 
community coordinator, the team Teitier, intewis^, and secretaries 'Tties^. . . 
people are charged vvitfh varied ^responsll>ilitie&j.and^ t2^s/lndicatofs'. of t • 
success among this^grqUp inclgcfe: identified r^eds 'or fejijEfy School pW-. ^ 
^ sonnel; solutions from research and cfevelopment to meet, loqally. identified V- 

; ne^ds; inservice activities plahned^and delivered to%neet identified nieeds>. ' ' 
ratings by Riley School staff ccrftmntng wejj^ effectiye^^esis^.a^ resburceV 
personnel for identified schoor^r^^ prblilems; l^atte^hdance at - \ 

, school .fufictions.>nd activities; ^racj^litatfen ♦bf* c6mmunicatid&>. among V- 
Variojjs levels of educators in ' tha^ locah ^ciftional conrjjpunity, - . 
documentation and assessment t^' in^ervicea^ct^S^ 

; School; documentation .ajfel assessment; of ain^ instru^fiphal 
'activities at Riley Schpol, In Leoijj ^uqty; and ih natioQaVTeac(ie^^^ • 

• activities; documentation and as'5essrnefTt,,of ongoing >nd sujGcessfuf'*«irf.- 

; •.fusion of programs and activities gen^ated ^t Riley^chool through B<ftfj . 

. L'eon County inservice programs and'the feachereSucgatton^prbgrams at fSw'^' 

" Riley School Community Members^ - " • , " * I - 

' ' . ' • - A '* ' j ' T 

' Riley ScKopI community nrifembehs iricfe^gJSoth ^gtents of students- lo 
: .. /Riley School and other members of * th^^;con*punir^v^^fe^ not?^ 
. c h I Id r e h . i h * att e nda il ce . S u c ces s H nd aiors >V irh th 1 s roi^T 
■ tendance' at school sponsored activities; .participatron ;in 
. voiuhfeer program; responsiveness to scihoel Reeds^amf r^Que 
'. arid partici:patiorr in the Rjley School Community C/^cWa^ 
. ' interest inventories. • * ^ - ^ _ ' 

Preservice and lifservice Teacher Trainingyins&t^ # 

. . ; This group includes the Colleges of Eductitiori at FSU an 
Leon County TeatherJdLrcation Center. InHicai^r^Qf «;iiWp<; 

' include: Increased intaj^tion amonejpducators arriong.thes^jyjree^iroubs:^ . 

' joint-planned inservice aft^jties.inv ajl thrle inaikkion^Pviderlofe of y 
diffusiorf into instructionl^rograms of activities, prppEns, an^khowledge 

. generated through interaction at Riley School, attendance. 'at planed in- 
service activities; ^_nd rated effectiveneS:S and r&levance of i|nservi(^ ac-'V 
tiyities sponsored By these groups at Riley Sghddi^ '4^ ' ^ 
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• GUSSBORdrCAMDEN PROJEt^ 

' • ■ : ■ ■■• •■' , ■ ^ . . . . ■ >.■ 

• o ^'JProject Diifector and Evaluiitor 

Frank Coodfellow, Directoi ' -^ 
; ; Geprge Brent, Prograni Dfeveloprnent^SpeGj^ arid Evaluatpr 
j^W;^lassborp State Col|ege/Gamden Gitf public Schools 

lenth Cycle Teacher Corps Pr^^ ~ . _ 

•' vDeparlrnent of Elem*rjtary Edu^^^^^ ' 
Classboro State College^ 
. ; Classboi^o/ New Jersey / V 

> Obj<sctives 

* ■■ , ■ ■ * • ■ ■ , 

1. teachers will be aWe to apftly Precision. Teaching in their classrooms. 
; '2. Teachers will be aiye to teach Precision Teaching to other teachers 
: and parents. / ' ' - 

3. Teachers will be able to serve as specialists in self-chosen 

educati<>hai areas. ^ 
4.. TeacHejrs will be able to advise other teachers iq the management 
^ and education of children wjtfHearning and behavioral problems. 

Description of Research lnterv«ntionV 

In keeping with the Tenth Cycle^Teacher Corps policy of baling each 
project Of! a research strategy, Classboro State College and the Camden 
• . Public Schools havechosen Precision Teaching as the tool for measuring the 
;V'. , performance and learning of children they may be affecting. Encompassed 

■ y in Precision Teachjr^g are strategies for pinpointing performances for 
i^"^ -^^^'^^^^^ "^ 3 direct measure of leading. Pinpointing 
:^l£fi-.P®'*^°''."^3"S^^ insures individually defined curriculunj goals for the chijdren/ 
ri^>>oritinUou^:nieasurement aljlows learners and teachers to make i^«f)id in- 
|J^^;^tructional^ The cfecisions are based on the as^55ed* amount of 

occurring for any pne pupil;. And last, but certainly important, 
is a series of techniques which allows learning data to be summarized 

As stated above. Precision Teaching offers a mi^ans for^ measuring 
Jearning ditectly. The direct measure of learning enables the project to 
* discover what forms of learning are caused by different types of teaching. 

■ The Classborp-Camden T'enth Cycle Teacher Corps Project is using the 
: ^ , results of the learnimg hrieasures to build the content for teacher training. 

Teacher trainers are fiftiding firm positions from data-based decisions to 
share with others. Alost important, there is. an accounting of how the pupils 
are growing. When pupils are not learning, it js known to them, to their 
teachers; and to the inservice trainers immediately. 

Selecting Precision Teaching as the Cla.ssboro<:amden research strategy 
was not by chance. Rather, selection of this research strategy was based on 
the availability of hard data. These* dat£» show distributions of "power" skills 
for childr'en'. These skills have criteria for syccess and learning measures 
which allow any learner to know where he or she started and where he Or s>ie 
4 is daily, * , 
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Within the duration of this cycle, the ClassborjD-Camdenjj^each^r Corps 
Project will be able to share: ' r 

1. what match objective sequenice works best /«piven pupils; 
• 2. what.proficiency levels of performances (jwouencies) are best for 

next level objectives, and ' ' ' 

3 what tool skills are critical for all learning, ^ 

Description of Target Population ' / 

..'**- ( : 

Precision Teaching is being adapted in the Forest Hi|l Schopl, Forest Hill 
is> located in CarTxler^ New Jersey, and vhas 16 regular and% transitional 
classrooms involved in the project. Most teachers are using Precision 
Teaching during mathematics instruction, at present, but will expand to 
rerading during the second year of the project In addition, the art, gym, 
science, library, and music specialists are involved within their subject areas. 

Description of Success Indicators 

Two types of measurement data will be used to evaluate the ef- 
fectiveness of the ^acjaptation of Precision Teaching. The first is the in- 
formation displayeji by the daily records of Precision Teaching. This in- 
formation wilJ summarize student learning improvement in each curriculum 
area, as well as effective teaching procedures and decisions for each student. 

The second evaluation measure will compare pre- and posttest scores 
on standardized tests given the experiiriental and control group^students. 

The goal of ^this Teacher Corps Project is ;to focus classroom 
m^asure^Tient into proper perspective as a dtecision-making tool to ifnprove 
teaching effectiveness. 



/ 
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MICHIGAN STATE UNIVERSITY PROJECT 

^. Names of project director and evaluatdr ^ 

" ■ • ■ . • ^ . '• 

Lonnie D. Mclfityre, Co-director, Michigan State University 
Minnie L. Wheeler, Co-director, Lansing School District ^ 
JoeL. Byers, Program'Evaluati6n Specialist, Michigan State Umversity 

Office: 20VG Erickson Hall, Michigan State \j^iv^;||iHy^A^ ' 
East Lansing, Michigan 48824 V 

• ■ ... ' ■ -'hM \. ■ 

DESCRIPTION OF PROJECT GOALS; ^^^^^^^^ 

Tenth Cycle Teacher Corps at the'Lansing SehooJ District and-Michigan 
State'University has as its foys the systematic i^daptatiorl of research fin- 
dings on teacher education. The entire staff of Wainwright Elementary 
School — a local Lansing school — will be irwolved In this program along 
with a team of teacher educators from MSU. _ ' 

•^Developmental teams will peruse tKe ^researc-h in the following 
curricular areas: reading, math, ngulti-cultural education and social- 
emotional, education. A fifth devetapmental teanr. Foundation^ of Teacher 
Education, will be investigating "VVfiat are the needs of society?" as they 
relate to the four curricular area^ previously mentioned. The efforts of the' 
foundations Developmental team will-lead to a seminar in which, all 
iljembers of Tenth Cycle Teacher Corps, Including the Curricfular 
Developmental team, will be participants. We feel that participation in this 
seminar will insure consistent goals as each deveioprrlental team attempts to 
build and adapt new curriculum\models based pn research findings and 
societal needs. 

The membership * of these developmental teams is comprised of 
preservlce teachers, inserVice teachers: and teacher educators — thus, each 
team \VHI t>enefit from the experience of the classroom teacher as well as 
input from the research efforts of the teacher educators. These develop- 
mental teams began to operate during Fall term, 1975. They will continue 
their developmental efforts throughout the school year with the expectation 
that curricular adaptations and instruction will be implemented during 
Winter and Spring terms. 

Preservice teachers will receive instruction in ail four curricular areas 
while inservlce teachers have the option of receiving instruction in two 
curricular areas. AM Instruction will be provided by^MSU, and the classes are 
to be taught at Wainwright School; A sp^ial group t>f seniors called 
"university interns" will relieve the teachers in orcfer that they may attend 
classes while at the s^me tim§ it allows these students to complete their 
student teacning requirement. Academic credit is available, and it may be 
applie^d toward a degree offered at MSU. 

Ai^ivities duringvthe second/year of Tenth tycle '1'eacher Corps will be 
essentilyyy^the same as the first year with the exception that the demon- 
strationM'ategy will be implemerited. Our project views the demonstration 
strategy^ one of three possibilities' — process, product or teacfaing. For 
exaniple, a visitor n^ight wish 'to .learn how a classrobm teacher becomes 
activelyj^volved in curriculum adaptation and revision — a process; 
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another might wish to see a new math Wrriculum for grade four based on 
research and societal Vieeds — a product; still another might wish to see i 
teacher th^it has been retained,using nevVly acquired skills based on research 
and adapted for her fourth grade math class — demonstration teaching! In 
addition |o the training an^ retraining ,corr>ponents, Tenth Cycle fl'each^ 
Corps win also, provide training for ai.des and parents iq the same four 
curricular;areas« \ ' ' _ • i . 

The Community Based Education Component has three major goals: 

—to orient Teacher Interns to the community; ^ 

—to familiarize the community witVj the Teacher Corps program; * / . 

— to rfecrult and develqp community groups desiring training in one or* 
all pi the four curricglar areas offered. 

If we are successful in achieving the third goal, i.e., the recruitment of* 
community groups for training and; rn turn^ are' able to provide the training 
in the currjjpular areas, then we feel this-* process will have a very positive * 
impact upon some of the children in Wainwright School. It will also involve 
those parerlte in a very direct way with the education of their children. 

The Exceptional Child Component will utilize an individualized ap^^ 
proach. A MSU faculty member with special education training will focus onT 
one child with a perceived problem and work with the teacher, the interns, 
the tecim leader and the clinic professor to build and develop strategies' 
based on curjcent research to ameliorate the problem, jSince this project is 
housed in ohe'sthool, it will allow more child/en to rec'eive individualized 
help from.a team of educators while at the same time it provides preservice 
and inservice teachers with sQ,me of the skills for mainstreaming. 

Data collection, analysis and evaluation will Ije^under the direction of a 
team of evaluation specialists from the Lansing Schdol U^strict and Michigan 
State Univiersity. A member from this team will serVe on each of the 
development teams to insure tf^at evaluation activities are performed by 
traine<l specialists. Every aspect of Tent b Cycle Teacher Corps is to be 
evaluated and T^ports that document sucfcesses as well as failures .will , be ' 
made available through the Research Network 'to other Teacher Corps 
projects. J. : ; 

Description of Research Intervention « 

The Michigan State University-|_ansirtff|^chool District Tenth Cycle 
Teacher Corps pr<3l^ect takes its point of*^partur'e for its research in- 
tervention from the senseof powerlessness that many teachers seefn to have 
In the day to day conduct of their pjofessional lives. When it' comes to 
determining what and hovy to teach, many others (administrators, school 
boards, parents, teac|\er unions, publishers, university professors) seem to 
h3ve much more contfel ov^rdecision-making than the individual classroom 
teachers. The staff of theWjSU-L.S.D. Teacher <Iorps program has 
developed an inservice training program a^ed 'at the amelioration of the 
feeling of power I e stress. This interverrtion, if* successful, will provide the^. 
staff at the target elem?litary school in the Lan^mg School District with thi^ 
technical and intellectual skills to make a^^e a^id^ just curri^ul^r- 
development decision,, the political skills to affect a major influence on^ 
curricular decisions,a'nd the will and commitment to us^ these skills. 

The, sense of powerlessness just averred" to springs not only jrom 
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psychological, sociological and political constraints placed on teachers and' 
. teaching, butin nri|r>Y instances to limitations in the content and knowledge 
of teacher^ Without increased knowledge about subject matter -and con- 
fid^nce iftAeir fcon^fjetfence of such matfors, a prograrfi runs the risk of many 
educatioTl^f programs"; i.e., being too abstVact. l! has been decided to 
partition instruction into four curricular area>VQf concern to elementary 
school teachers ahd to the national Teacher CoV- These four curricular 
area^are Reading, Mathematics, Social-Emotional Education ^nd Multi- 
cultural Education. If teachers are.to learn to overcome the sertsp of 
powerlessness, it is important that ready-made inservice proij'^ms not be 
"laid upon them." Therefore, four curriculum development^!&ams were 
organized during the fall of 1975, one for each of the abov'^mentioned 
curricular areas. Each team consisted of several university faculty members 
whose areas^f Scholarship^ centered in the subject matter area. In addition 
to tf^ teach^ educator, each team had a regular faculty member from the 
target elementary school assigned to participate fully in the development of 
the ins^wae>instructi^^ objectives\ and plans which constituted the 
research irjter^ention of the Tenth O^cle M.S U -L.S.D. Teacher Corps 
Brogfam{ f : * * ^ ^ ^ 

■** ^ ■_■' '■> ■ \ > *' 

Descriiftiori of Targfet Population(s} 

;ar 20 Inservice teachers at Wainwright Elementary School 
. ^ 2males, 18 females; 1 black, 19 white . 

b. ) 4 Teacher Corps Interns , 
/ 4 females; 2 blacks, 1 Latin, 1 white , 

c. 2 Team leaders / . . 

1 black, 1 white ' , 

d. .1 Principal , ^ 
%^ white ferhale ■ 

e. "Elementary Students ' . 

576; 22% minority (blatk and Chicano) 

(basically Iqwer middte^rlass - 8Q% with some working 

class— 20%) ^ . ' ^ 

f. 22 Pr'eservice Students ^ » p 

2 m^les, 20 females; 1 black, 21 white ' . \ 

g. 12 Teacher Edudators ' . " ' 

5 males, 7 females; 2 black, 10 white ' 

h. 12 Xeacher Educators in Training ' .. \ - 
^ 2 males, 10 females; t black J Chicano, 10 white 

Since th^ rriajor thrust of the M.S.U.-L S.D.' Teacher Corps Program is 
directed at tepcher curricular devebpment and decision-making, it seemed 
appropriate, trtat the majof dependent,variables relate to teacher behaviors - 
.Each curncul* development team was charged with the construction of a^ 
set of evaluation items. which would fall into the following three categories- 
1) whA are teirhers' self-perceptions of their curriculum planning'iskills and 
their valuing of \ch skills?; (2) what is their actual knowledge of curricular 
content and curricular plarining skills?; and (3) what is the extent of fheir use 
ot these skills ip their teaching and planning? These three questions flowed 
tfom the project's commitment to changing the sense of powerlessness by 
providing . teachers with the technical and intellectual' skills =^to make 



APPENDICES , ' !f h 

\" ■ - % -■ '/■ 

curri.cular decisions, the political skills to influence th^ decisions^^nd the 
commitment to use thes^ skiMs. The first two of .^se questions were 
concretized in a set of foii^eyalu'atipn instruments (one for each curricular 
area). The third questifl/h dealiiig with the extent to which teachers actually 
employ decision-makmg^ 'and •curricular, development skills^ in their 

^ professional lives will be Answered throqgh classroom obse^ations. 
' In addition to this plrimary data source, the Lansing School District has 
made available to the Tenth Cycle Teacher Corps a complete data base of 
pupil achievement for the target Teacher Corp^ school as well as several 
other similar elementary schools in the district/ Although it is not Expected 
that students will show Ah immediate- change in performance on the stan- 
dardgtest pleasures included in this data, file, it is expectedothat over the long 
term as teachers gain in competence and confidence in curricular decision- 

^making and development, that student scores will show a systematic im- 
pro,vement; We therefpre hoptf to track the performance of the students at- 
the tsurget school over an extended period pf time. 
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MURRAY STATE UNIVERSITY PROJEa 

Ptoject Director: Dr. Jerorne Hainsworth 
Prdject Eyaluator: Dr. Bill Price 

College of Hyman Development and Learning 
.A s Murray State University . 

Murray, Kentucky 42071 ^ ; . 

Project Goals 

• This project proposes to create a teacher-student learn^ing situation that , 
will identify, and maximize divergent lelirning potentials and styles In both 
teachers and sj^udentstotheend that each child will be properly classified as 
"gifted." » ' 

The primary ol^jegtive^^of the project are: 

1^ Identify talent strengths as identifiejd on tests of academic, creative, ^ 
. planning, communicating^ forecasting and decision-making abilities 

2. Developing organizatibnal strucliJres to focus a wide range of efforts . 
on maximizirig and developing identified talent strengths. 

3. Developing teaching strategies whicti show the greatest promise for 
each teacher, and intern developing stucfent -talents. r^- 

4. Integrate teaching activities *with: a variety- of m^ans for 
syste>natically gaining infprniati.bn' on students' potentialities and 
progress. . ^ ; j " j 

Research Adoption - ' ■■■ ^ o 

There exists a substantial body of research that indicates most children 
possess some specialized ability of ^.high o'rder if enough ways were 
^available to test for thbie a^lities. That each individual child c|oes have 
. unique talents is critical ^t^i this project.. The purpose of this project % to 
. create^ within the School :a unique atmosj^here where the teacher has the 
opportunity and has developed the skills, to v\ew and discover the 
uniqueness of-eaCh child. 

Bloom's stJccinct statement expresses the project's* purpose: 
The consequence of teachers' viewing each, individual 
! as possessing unique talents of a high order (in contrast with 
viewing only a small group — perhaps 10 per cent — as 
having a high generalized ability) are/^uite pr6found.\The 
teachers' appreGiation of the unique *^merits of eafch. child 
could have ini^rtant consequences for the ways in which 
the student a6d teacher interact as well as.for the ways/in ^- 
which the teacher niight try to help each student in th^ 
learning^ process. , ^ ^ V 

" The purpose is focused upon the de^nstration- reauTrement ;pf th6 . 
training for systematic adoptioiftof research findings ahdTise«^nstruQtiSnal 
and organizational strategies <»human and physical resources to c^mSrf 
stra teethe ad apt iveness of reseajK:h. ' 

j \f\ brief, it is to demonstrate the practic^ity of the research which tia*> 
found almost all children to possess unique talents of a nigh ord<&r anf(^ 
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^capable of being called ''gifted" in some ^rea of their; development. This" , 

!projett will apply the research in this area in a middle school teacher-student ' 
learmng situation in which low-income and min6ritY studjents comprise,- 
apprbximately one-third of the student body. Th'fe project, will also 

^demonstrate the type of teacher training and retraining necessary to be* 
pVovided by teacher training institutions for j^eeting the problems rels^ted to ^ 
students of low-income families. ^ ^ . . 

"Gifted" is defined in this proposal as a talent or ability' a person 
possesses that can be identified and developed that is of a unique and high 
order. This talent may be one that is not/evealed in a gene*ral abilities test or 
m the standard norm tests that are administered with a single result reported. 
It will therefore be necessary to administer ability tests and other pre- - ^ 
assessment instfuments, both commercial and teacher made, to discoyer 
divergent ways of classifying' and developmg special unique talents that 
students ^HDOssess. It should be ^emphasized that identifying and maximiz- 
ing each child's uniqueness is not a siniple matter of test q>nstI"uction and 
administration. The program being developed is a unique student-teachet 
interaction process, that could have significant educational results. This 
interaction is based on totally new concepts for experienced teachers who 

» have be^ teaching on the previous assumption that students fit the bell 
curve of low to high achievers with the largest percentage of children in the 
middle. The new concept that Bloom and other researchers have demon- 
strated is that each child has some ability that is^sefui and important that is 
at the high or gifted end of the capability scaljp. Specific training^fr) human 
potent!^ as well as diagnostic prescriptive and evaluation techniques are' 
requii'ed for experienced teachers as well as Kacher interns to demonstrate 
the research application to students of low\ncome families. 

Project Research and Evaluation Activities 

Research and evaluation activities within the project focus upon the 
measurement of impact in at least four dimensions: (1) impact upon 
students, (2) impact upon teachers, (3) impact upon the school and its 
learning environment and ,(4) impact upon the community served by the 
school. The assessment program endeavors to measure more than student 
performance alone, although academic achievement levels will be examined* 
through analysis of C.T.B.S. scores. A comprehensive and realistic measure 
'of student outcomes must include measurement of key factors descriptive of 
conditions and educational processes which influence student performance. 
These include factors such as teacher attitudes toward a variety of 
educational concerns and processes, as well as measurement related to 
degree of student classroom participation, teachei; emphasis on rein- 
forcement of self-concept, levels of individualization within the classroom, 
and development of potential in unique talent areas. 

The basic concept around which this project is organized necessitates 
the generation of data of at least two basic types: (IKidentification of 
specific and unique talents for each student, including the normal range of 
academic competencies determined by the C.T.B.S., and special potential in 
the areas of forecasting, decision-making, creativity, planning, com- ' 
munications, and leadership, and (2) descriptions of each teacher's in- 
structional environment — educational process data — as perceived by 
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Students within that environment. Talent identification data and educational 
process data both become input for the inservice instr.uctionarprQgram as 
teachers develop, refine, or modify instructional systems, materials, 
management systems, organizational patterns, and irtteraCtion 0atterns 
within their individual teaching egvironments; ' ' . - . .* 

' , Educational process data is coHected primarily through the 'use qi. the 
Student Activities Questiorihaire (SAQ)i a 65-item instrument which' focusfes 
upon studerxt peVceptions of their learning envkonrtient. The use of studenta''^ 
perceptions as oppol^ed to available observational/analysis protesses for the 
collection of these data offers one qlear feJvantage: process data a>;^ailable to 

. project staff, interns, and teachers is provided in both ndrmative and on an 
individual student basis. - ' . 

Since the school itself can be'cqnsidered as 3 single environriFient,'it was- 
deemed essential that specific 'data Be generated that teachers- ^nd ad- 

^ini*rators'm.ight use in decision-maklftg related to the to^Uchool and its 

/program^^To act and rea^J Intel lifeently, every member of.the'schodhcom-' 
munjty needs to kndw.hbW otKer members perceive the realities of school 
life,' of the environment of the school - t)^ laterplay among its people, 
processes, and resources. Important dimensions of the^enyirohment include 

' the wayoeach individual, feels about himself and the perceptions, values, 
expectations, satisfactions, and dissatisfactions* of t\]e yJ^riqus groups that 
make up the school community. To enable the/' school faulty aTid ad- 
ministrators to have and^use this kirid of feedbael?, data was ^Stered using 
the E.T.S. version of Queijta—"Questa land Questa 41 

The basic research fmdings that this project has identiiMsM tha£ 
unique talents in children can^Be identified and developed. TTwe^f^hasis in 
this stateiTient is oo the words developed or enhanced as well as ideni 
tificatron. This project's goal, is not just to recognize uniqueness of in- 
dividuals of a high order but to provide a diagnostic analysis and a positive 
prescriptive program that yvill enhance the id^tified abilities. 
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- ^ ' OKLAHOMA PROJECT 

\ The Oklahoma Teacher Corps Projtect is located at Moon Middle School, 
190TN.^.;1Jth,Oh4ahoma Citv,. Oklahoma 73117. The project director is Dr. 
Thomas H:'GairaHer\whose office is located at the University of Oklahoma 
campus, 555 Constitution! Avenue, Nomrian, Oklahoma 73037: Mr. James A. 
Smithv'^Prcteram Develofjmen^ Specialist, is responsible for coordinating^ 
.evaJltftk)n^ictivltiei and Storing and retrieving documentation data. He may 
be addressed at the project site. ^ . . 

The Ol^fahoma Jeacn^r Corps Project was designed as a demonstration 
center which woul<l;promote professionaj and^ personal development among " 
educjatfonal worke'rs of yariqus role grbups through the implementation of a 
trgihMTfg compleai^ In^upporting this major purp'pse of educational personnel 
development, the project\v>employs, at virtually every level of,, project 
operations, a collaborative d4^cjsion-making strategy involving persons- in-. 
institutj(fJhT\of* higher education,^ the public school system, tbe target 
¥:onimunity tdt)e>efV6d,4S well as those persons at the project site. ► "1 

The threw major program components in th;^ project are; Training for 
Inexperienced Teachers, Tpanning for Experienced Teachers, and Community 
Volunte^ Trying. The major thrusts of the training cornponents are 
^ categorized asTollows: (1) Multicultural Education, (2) Community-Based 
Education^^(3) Accommodating Exceptional Children, (4) Study of Teaching,* 
and (5) Study of Organizations. ' 

Project goals |nclude the following: , ' 

1. Througfi a collaboratrve effort institutions of higher education, the 
local education agency and the community served by it, will devefop' 
' -- a plan for continuous implementation of a staff development/ 
^ ^ proc^ss^nd .the expansion of the cdncept districtAvide. 
^ * 2. To dev^p procedures for planning, implementing^ ccfordinating and 
r evaluating the Teacher Ciifrps -Project, m i * 

^ 3. To establish ^ training complex at a school site through which 
personnel development programs will be delivered to groups arid 
individuals.; \ 

a. Program Components 

1) Inexoerienced Teacher Component 

2) Experienced Teacher Component 

3) Comniunity Volunteer Component 
\x Training Thrusts * * 

1) Community-Based Education » ^-z^^,'' ' \ \ 

' ' 2) Competency-Based Education - 

3) Accommodating Exceptional Chjldrert 

f 4) Multicultural Education * , 

5) Study of Teaching ^ . 

6) Study of Organizations - ; ' i. . 

4. To integrate the* resources of Teacher Corps tlocal, network, and 
national) and those of the community and institutions of higher 
education in meeting the needs of educational personnel at tl)e' 
project site. ^^|r ^ 
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5. To improve the learning environment and experiences of the student 
poputation to be served. 
■■• <■ ^>''." ■ ■ , / ■ ' • . • • 

: 6. To establish an effective system of communication for the Teacher 
. Corps Project. ' " 

'■•■>-' ' ' ■ * 9. « , . ■ . 

, • • a. Internal Information System ^ ^ 

'■"f . 1) Management •*ream ' ^ , 

; > . 2) Project Steering Cohimittee 

3) Staff Advisory Committee - > 
, 4) Community "Advisory Council 
• 5) Project Staff , ^ » . 

. b. External information System ' 

1) Board of Education/Public School Officials . 
V / Department of Education 

3) Universities 

4) Community , 

5) Teacher Corps (relational, Network) 

7. To monitor progress, assess project effectiveness and disseminate 
results to relevant publics. 

8. To assun^e responsibility for Teacher Corps 12th Cycle proposal. 

The project's major intervention strategy is the establishment of a 
training complex to facilitate the implementation of a continuous] personnel 
development; process at* a school site. All training programs will be 
developed based on needs identified by target groups. The delivery modes 
vylJI include group and irtdividualized/personalized instruction in a com- 
petency-based format. Each tr'aining program will incorporate aspects of the 
six training thrusts listed above. , . 

Target f opulation 

, The major target population to be affected by the project is the faculty 
and staff, of the Moon Middle School, ehe target site. Figure 1 reveals a 
profile of the faculty and staff at Moon. 



Experience Highest Degree 



Race' f 


Sex 


Age 




Black 26% 


Female 56% 


20-25 


14% 


White 56% 


Male, 40% 


26-30 


31°/9 


Other 9% 




3i4o 


26% 






41-50 ^ 


14% 






51-60 


9% 






Over 60 


2% 



1-4 yrs. 


36%* 


5-10 


34% 


11-15 


9% 


16-20 


12% 


Over 20 


4% 



Bachelors 53% 
Masters 43% 



Figure 1 
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The data indicate a relatively youthful, somewhat inexperiencecC 
minimally licensed, multicultural faculty and staff who, during negotiations 
with the school district for a possible target site, requested that the project 
be located at^qon. The request alone is significant evidence of a faculty's. 
abilitV to recognfee re-training needs and the potential resources oS Teacher 
Corps in *sisting in their professional "development. 

the, administration and^f acuity at Moon are, seeking; to implement team 
teaching, improve instructional skills and interpersonal relationships^ 
develop better communications, provide more meaningful educational 
experiences for* students, improve faculty morale and student attitudes 
toward school. « ^ ' ■ 

Of secondary importance as target groups to be affected by the 
Oklahoma Teacher .Corps Project are: (1) the student population at Moon 
Middle School, (2) the community served by the school, (3) institutions of 
higher education* and (4) the school district. • , 

The Oklahonfia City school district is cun-ently operating under a court- 
ordered desegregation, plan which involves the cross-town bussing of students 
to achieve pre-determined ethnic. ratios, t^oon Middle Scjiool is located in a 
community prec|pmlnantlv populated by low-income, black families in\he 
east-central part of Oklahoma City." Under the desegregation plan, the Moon 
student bo!dy is comprised of 1,040 students, grades 6-8, with approximately 
33% of tKe student body Identified as poyerty-stricken. The ethnic mixture 
of the student body is 32% black, 54% white, 8% Native American, 1% Chicano, 
and. 1% Vietnamese. 

A needs assessment study indicated that studerit suspensions for rule 
infractions were high at the schtii^, achievement was relatively Ipw, and that 
even with the existence of special education classrooms, and public support 
services, a large number of students with special learning/behavioral 
problems were being neglected. 

While the school mainly serves the east-side community in which it is^ 
located, the desegratlon plan requires students from other areas of the 
metropolitan district to be.bussed to Moon. Thus the^community actually 
served by N;1oon Middle School becomes virtually the entire school district. 
Community-related project activities, then, involve parents of students from 
approximately' thijteen "feeder schools." ' 

; The Oklahoma Teacher Corps Project Is a con sort lum^gbveTrned project. 
The OMahoma C6h«prtium for Urban Teacher Education was formed in 1%9 
and currently consists of the following institutions: ^ • ' 

Oklahoma City Public Schools .* ^ ' 

Bethany Nazarene College , / 

Central State University 
Langstpn University 
Oklahoma City University 
Oklahorpa State University 
Southwestern Oklahonr^ State University 
University of Oklahoma 

The project also has as its target group these ^institutions of higher 
education by Influencing their programmjitic efforts t6ward competency- 
based teacher education, the exchanging of personnel for consultative and 
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instructional ()urposes, the mutual sharing of resources and other support 
services. Thys, the project seeks to influence collaborative efforts in 
teacher education on a. state-wide basis. . * 

Assessment Instrumento 

To provide baseline ^lata for futile comparison the following 
assessnnents and measurem^ts were administered to Moon Middle School 
faculty and staff. . , ' 

• '\ * ' ' ^ 

Minnesota Teacher Attitude Inventory. The MTAI was ad- 
. ministered on a pretest/f^osttest basis to determine or dif- ^ 
terentiate those teachers who held jK)sltive attitudes toward 
children and their profession, jfie celi^billty of this scale for . 
25 ratings was .93. 

Oi||anizational ClimaU Description Questionnaire. The 

OCDQ was administered on a pretest/posttest -basis. This 
instrument provijded useful infbrmation, describing the 
organizational .climate ofVthe school. The instrument was 
developed by Andrew Halpin. . • 

Teaching Different Pupils. This instrument, developed by 
.D6rothy Skeel, was used to discriminate between iteaehers 
with positive and . negative attitudes tow&rd dilturallxL 
disadvantaged children. . > T 

Staff Jnlomiation 'Inventory. JWs'informatibn form was used 
to provide a profile of the staff at Moon Middle Sghool The ^ 
instrument was locally developed 
* • • 

Moon Middle School* Needs Assessment. The information 
obtained from this instrunjent provided the staff with a list of^ 
ne'ed indicators from which criteria variable^ could be 
determined. .This ihstrument was district constructed. 

Course Interest Survey. This instrument was used to further 
vahd;ate the needs assessment information on priority 
training needs. Tbe'survey was locally developed. ' 

Workshop Evaluation Questionnaire. T^e purpose-^ of this 
instrument^ was to determi^, the effectiveness of the 
preschool workshop in attkl>»ing its objectives J of <^ (T) 
establishing departmental goals, (2) determining policy and- 
' pj providing baseline information. The questionnaire was 
locally developed. a ^ ' 

16 provide evaluation data for future comparison changes in 
• student achievement, attitudes, perceptions, and behaviors 
thp following information was collected. To study gr^de 
equivalent gains, a random stratified sample of one hundred 
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students from ^ach of three educational levels (grades 6th, 
7th, 'and '8th) was administered the Metropolitan^ 
Achievernent Test on a pretest/posttest basis. This battery is 
, * widely used among schools^ajicf its reliability and validity is 
relatively acceptable. -The use of this test by the Oklahoma 
\, City Public Schools make its selection for^ the project both 

' desirable and expedient. 

•> . • 

Grade Subtest ^ • . Form ' ^ 

^ 6 ; ^ ^ Word Knowledge, >^ ^intermediate (Form C) 
Reading Language, * 



1 



Spelling 

Word Kr 
Reading 
Spelling 

Word Kr 
Reading 
•Spelling' 



Word Knowledge, " Advanced (Form F) 

Reading Language, - > ^ 



Word Knowledge, , Advanced (Form C> 

Reading Language, 



Learning Environment Inventory. Thi^ instr'ument Was. ad- 
ministered to a random sample of 100 students ifrom each of 
three levels. The irfstrument measured students' perceptions , 
^ ^ of the learning environment in their classrooms. 

The Children's Self-Esteem Inventory. One hundred students 
^ from each gf, three grade levels were randomly selected end 
administered this mstrurfient. The Oklahoma -Klity Self- 
, ' .Esteem Inventory has a.test-retest (stability) correlation of 
.+0.88, and content validity based on an expert panel of Title 
I teachers, consiiltantiL and administrators. Concurrent 
; validity data reveal a coprelatipn* of .706 with math com- 
putation ^ ' ^ C 

Student ' Population Information was' obtained from the 
schooldistricf s Department of Research and Statistics for the 
purpose of proposal writing. Jh\s information was revised as 
the project began to obtain a more current arfd accurate 
. reflection of student populatioh characterist^csX 

Attendance and Suspension Data. Information \egarding 
these variables was obtained from^ the school District's 
Department of Research and Statistics where this data. is 
continuously gathered and stored.. 

. ■ ■ 

Referral/Placement Ratio. Data regarding the number of 
^students placed m special learning programs after being- 
referred for diagnosis are monitored from records stored in 
the school couriseling department. ^ \ 
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SAN JOSE STATE UNIVERSITY PROJECT / 

Director: ' Robert L Spauldihg " ., ' * 

Evaluator: Bruce Joyce * . 

Address of Project: ' . * " / _ 

• Room 20O, School of Education ^ 
^ ; -San Jose State University, Saa Jose, CA 95192 • 

Telephone: (408) 277-2666 ' ' * 

Project Goals ^nd Objectives 

The . principal goafs of the San Jose State University Teacher Corns 
Project are: . ^ 

1. To.utjiize the results^ of four different lines of educational research 
(teafcher selHnalysis, models of teaching, pupil learning styles and 
teacher decision making) to assist inserviee and preservice teachers 
, ■ • to develop skill jn studying and improving the teaching arrd learninc 
that occurs in their classrooms. 
2; To provide teachers with optiorfs to select from an array of training 
possibilities as well as. to generate thei/own plans for training ex- 
. per4ences in order to meet'tldeir training needs as completely as 
possible. / 

3. To integrate, and therefore, adapt more adequately the findings* of 
•several related but separate lines, of research on teaching- 
• sP^c»ticdllyyo relate tr^inmgin models of teaching to pupil 1^^^^ 
^. styles;and to relate botKof these to teacl^er decision making 



The objectives of the project are: 

• ^ * ■ 

1. To provide for each student those teaching/learning experiences that 
build upon his learning style and his particuJar learning strengths. 

2. To provide a ^teaching/Jearning^rogram that attends tb each 
jtudent's special culture. ^ . - 

; ^3. To build an information balse that attends to relationships among 
teaching repertoires and learTiing styles. ^ 

4. To involve parents in the instructional decision-making process. 

5. To train- parents in the use of instructional strategies that will 
enhance learning in the home. . • ^ 

6.. To broaden teachers' repertoires of teaching skilU through training in 
alternative models of tpaching. 

: A ■ ■ ' ^ K 

7. Tp develop a teacher observation, feedba/:Ic, and iself-analysis system 
that attends to a teacher's skills repertoire and the learner's strengths 
and learning style. . 
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style. ^1 p - , 

9. To provide immediate feedback to the teacher regarding hi3 per- 
formance for purposes of» self-anaiysi?. • 

10. To provide a cumulative recorcl^of teacher-learner interactions and 
_ their outcomes for purposes of instruclionaJ decision-making. 

11. To train four teachers new to the profes;5ion in skills related tos^- 
analysis, rnodels of teaching, adaptatfori of contemporarV research 
findings, learning styles, and instructional decision-making- to 
develop their capabilities for responding to' the needs of students 
from multi-cultural bagkgroynffS. . 



Description of Interventions and Tr^^^'ents 

r , ♦* ■ * • 

teachers in the Rogers SchboU are offered access to any one or any 
outlined in Section II The training components offered are these. 

^ Component I: Self-Analysis of Teaching 

' u ^t]'^^°'^P^"^i^*^^^^® training Pi'ogram .was desi^^^ 
u'Pi- ^^"^V'^'s/her interaction wkh a single 

child, who is selected by the teacher because of some 
learning problems. The teacher decides what teaching 
. procedure Fnight work best with ct he child, and also deter- 
i • mines what part of the interaction will be studied. To help 
teachers study this interaction, trained observers watch the" 
teadier and pupH .working together, and record what hap- . 
pens on tally sheets pr cm a small, hand-held typewriter like 
machine. Thesfe dataware sent over the telephone to a 
computer, an^ thjp. computer sends back a graph or some 
other easily-read drsplay of information for the observer and ' 
teacher to see. To help the teacher understand the computer 
display, the observer is able to "translate." As a result of this 
type of sti^dy, teachers identify which procedures-work best ' 
for particular learning problems (Morine & Vallance 1975- 
. Semmel, J972), . , , . . ' : '. 

Component II: Mo<)els of Toeaching ' 

Training in Various models of teaching forms a major 
component for both interna and experienced teachers at 
Rogers School. Each modtel of teaching is a teaVhing/learping 
strategy that has certain goals and a specHfic sequence of 
activities to achieve these goals. Each model has an in- • 
structional system that teaches the theory of the modtl ^ 
trains the teacher in the critical skills' of the model and 
provides feedback so th^it the us^ of the model can be 
rpastered.'^ . - 

Through the use of models of teaching, teachers can 
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desigf) a wide range ot learning environments. Some models 
of te0ching haVe group process as their emphasis,^others 
stres^ • personal skills like understanding feelings and 
creativity and other. models ha^^>developing tonceptMttKl 
thinking . processes as their emphasis." The work 0t BrO^ 
. Joyce and Marsha Weil, a^ expressed in their book/Models of 
Teaching (1972), is the major source for the trainings, 

C0fnponent III: Applying Corilemporary Research Findings 

This component of the training program was designed to 
• help teachers find out what is currently being learned by 
researchers about effective teaching. The J^ar West 
Laboratory was to collect this information abouf current 

•* research, and file it in order to makeyt easy to select in- 
formation related to teachers^ classroom? fepncerhs. Teachers 
were expected t6 identify spectfff }i0r\^j\s and seek 
assistance via the "research couhselor," Wkfe vyould consult 
the data file at the Far West Lab. \ Vc)^'^ V.;/ 

The initial plan is being , m6difi^: ' .Tfi#:. research 
counselor has been meeting other typjes of ;Sf)ec(fid^^ 
of teachers. Rath'er than seeking research findings, teachers 
at Rogers School are rn need of;, counseling related to 
children, parents, other teachers, administrators and * the 

^peacher Corps staiff. A fuller development and jlemonstration 
of this component is underway under Dr. R. Dershimer's 
leadership. - 



Component IV: Learning Style of Studenis' 

'As in the "self-analysis of teaching" component, this 
approach uses a trainerf observer. Teachers select pupils who 

* are not responding or behaving as well as they might and the 
observer takes careful notes over several days to discover, the 
pattern or "style", of Jearning that each pupil sfiows' 

V. After studying the classroom learning ^'styles" of the 
* pupils the observer shows the teacher the results of her 

* ^.obseryatiopir^nd they, together, work out a plan to improve 

the ]earn/ng situatic|i for each pupil. Changes in seating 
^ arrangements, ways in which assignrpents are made, ways in 
which disturbances are deak with, etc., are suggested and 
agreed upon. This is called the "planned treatment." > 

Once a plan has been* worked out the observer visits the' 
classroom regularly and keeps notes on each pupil's 
responses to the new plan. The notes are shared each time 
with the 'teachers and adjustments to the pkn are made if 
needed. The data gathered each day are recorded on a table 
(or a graph) so that the teacher can easily determine hoiV the . 
, pupils' learnmg s6/les are changing as a'consequence bf the 
planned Keatmehts In the classroom (Spaulding, 1971). 
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Component V: Teacher Decision Making • - 

, This component of4he project grows out Of the training 
of the other four components. For exannple, teachers who 
have had tralnirig in all four components will have: 

'1. studied their own teaching to learn which procedures / 

work best for particular leaming problepns. 
• 2. learned several new teaching procedures (models). 

3. gained Information from the research counselor to help 
^ them deal with their classroom concerns. 

4. Identified their pupils' Individual learning styles. 

They shouldl)e able to bring all of this information together 
: to rnake decisions about how best to work.with the various 
pcipils- In their classrooms. 

^ ■ ■ ■ , . . 

Component VI: The Exceptional Child Component 

The exceptional child progranv'S^ based on, the Iden- 
tification of each pupil's charatcftrfstic social behavior and 
' task orientation by means of close observation In the 
classroom. As a consequence of such observatl^, combined 
with consultation, with teachers, parents, and special 
education personnel "(as appropriate), personalized programs 
of classroom Instruction and , behavioral management a*re 
developed for ali children. These personalized,' 
diagnostic/prescriptive procedures do not * require the ' 
conventional labels of special education. Instead, teachers 
are assisted in making use of observation data to design 
appropriate educational envlronrnpnts suited to ^ach pupil's 
. own way of coping with the social and,academic forces of 

the regular classroom., ^ ' , - 

Description of' the Target Population . • 



K/imary emphasis ot training activity is upon classroom teachers arSt^/ 
the Interns. However, many garget groups arei)lrivolved. The dlscernable '^^ - 
groups and Individuals are: ^ v 

1. Four interns / \ . <u 

2. Eighteen classroom teachers 

3. Speech specialist v;, , \ ' 

4. Reading specialist^ ^ 

5. Teachers (2) of tHe educationally handicapped 

6. Team Leader* 

7. Community Coordinator 

8. Site Coordinator 

9. Principal 

10. Eighteen Classroom Instructional Aides ^ 

11. Parents In parent/child study groups 

12. Hlg>i school tutors ' 

13. University faculty In teacher education (20) 
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Descriptioii^ of Indicators of Success 

Success of training programs in the San Jose State Urtiversity Teacher • 
Corps Project is determined by examin^fon'of trainee change. Base line 
measures are compared with process and outcome measures of the following 
types: * , " * ' 

1. Involvement in planning and trainijig activities, ^ 

2. N2articipatlon In planned training activltiiBS. 

' 3. Aittitudes toward training activities as measured by the Levels 
of Use of Interviews (developed by Frances Fuller and Gene 
i Hall of the University of Texas at Austin). • 
4". Acquisition of prototypical behaviors vis-a-vis ^ch ih- 
i^tructlonal and classroom management model (as measure^ 
By thft T,I,S, observational, system developed by Bruce Joyce, 
and STARS developed by Robert Spaulding). V ^ ^ 
5. Changes in pupil attitudes, glassroom behavior and 
achievement data as a- function of training jevents fas ^ 
measured by Spaulding's CASES classroom observation 
system 'and by Rogers School achievement test records). 
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STANFORD UNIVERSITY PROJECT ^ 

-■./<",'.','• 

Prpje^tOirectdrf Det^cerAvafker / * , - • , 

Evaluatiort Urtit:. E.G. Begfe;^ Barbara Pence, Lynrie Cray, Iris Berke 
Address :^ ' Teacher, '^(Jrps ProjecV ' ^ 

. ^ • . • Stanford. Univecsity • ^ , 

• . . - Stanford, Cahfomja''94305 

. » ■■ ' • ■ ^ ^ . . 

Pibject Coals ^ 'r ^ . o 

1. To improve vth^ TeducBtion of children fro^m low income 
y • fam'iles. ' \ \ ^ 

2. To iraprpve the programs of preservice teacher preparation 
offered by Stanford University, ' ' . ' • 

3. To Irhprove programs for continuing professional education 
^ at Stanford University.* 

4. To determine thfe effectiveness of our design for applying 
research findings. v , - 

Research Interventions and Treatments 

'Each f omponent of the complex Stanford-tdoover project determines 
what researchr<trategies,.interventions, and tecnniquesit Will use. We, had 
zinticipated during the planning Mages, that reliance oh^ previous research 
results and willingness to implement s^^a^l-scale research projects ^ at the 
schooUite would be a roajor part of the project.^Although wfe have not been 
as successful as'we had expecterf.in using yaHdated tesearfch results, and In 
irtiplementing ourown studies', various conriponents of Ihe project have tried 
a number research interventions. . . 

jKe Matff Work/Study team fcondpcted a study to see how much and 
wh«it kinds 6| m^th learning is retained over the summer. This involved 
testing in the isjJringof 197?and again in the fall of 1^.5 to see what students'' 
retained. RestiJts of this study will be analyzed to help determine What and 
lipvy to\each so -that maximum math learning Is retained during vacation 
•* periods. 

The Language Arts /Work Study team has examined literature oh team; 
teaching, using^contracts, working in open space, and is now looking at 
effects o^ language Int'erference versus reading problems in children's 
reading Errors. This group plans to t^e oral re|ding samples of children 
identified by teachers as being poor readers aridj^or of a Spanish language 
family (the d^ign is not definite yet), e)(:afnln^. them for reading and 
language interfereoce errprs, and modify teachingUechnlques accordingly. 
The language art$ teachers also * plan to introduce training In sentence 
combining (CHare's Work serves as the guide), and compare students' 
composition writing ability before and after Work in sentence combining 
techniques. v ; *. ' . 

The Physical Education Work/Study team plans to adrninister a survey 
of attitudeSi toward physical education to Hoover teachers, Students,. and 
parents. Irt adcHtlbn, they :hope to adnriinirtef a survey, to flr>d put what 
subjects within P.E. the students and their parents want taught. When the 
results of this survey arcanalyzed, they hope to. be abi€^,to implement* a 
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teach certain skills or courses which the regular teachers do not know how to 
teach, such as MexiCi^n Folkdancing. This group has made great strides in 
implementing theic :sift;^ucatiional physical education program, \ahd has 
done much in developing jfohtract programs. Next year they will try a study 
on^hree groups (one wd^kWa^n totally independent contracts, oyi working 
the traditional way, ^^WSk third i^ing a combination of ytrfdepenci^nt 
contracts and the traditwin methocl). The group has use/ video-tapinc 
eqgipment to tape students swimming and cooing various yOfher-activitieJ, 
and has used the tapes to focus studenb^attention on particular aspects of 
their performances. Although) no actual controlled studies have been 
conducted to date, we have seen great changes in this gi^bljp, evidenced by 
^ the way teachers are conducting their physical leducation classes, and by the, 
organizational arrangements they are interested in trythg. ' 

The Social Studies/ Work/Study team has been working Targely in 
curriculum sub-committees; on seventh and eighth grade curricula, on a 
multiculturaLuriit, and on a uhit dealing with use of newspapers. - 

.Tbe Bilingual Education Work/Study team administered a Language 
Dominance Survey to 85 seventh graders who had been identified by their 
teachers as being from a Spanish-language background. Unfortunately, the 
test Was designed for elementary school children and although the students 
at Hoover Junior High have very' low reading scores, their knowledge of 
English i^ sufficiently great that the test proved to haVe too low a ceiling, and 
was an insufficient discriminator of . English-Spanish language dominance. 

The Multicultural Work/Study team administered an attitudinal survey, 
the Multi-cultural Climate Scales, to'all Hoover students and faculty, which 
is to serve as the basis for developing the multicultural team's priorities. 

The Community Involvement Work/Study team nierged with the 
Community Council, set goals, including the implementation of a tutoring 
program,-ahd developed, together with theEvaluation Unit, a questionnaire to be 
administered to the community. However, the principal would not permit the 
administration of this survey. » 

One of the. project's goals had been to preface work in -any component 
by an extensive literature search. Many components turned up little of use to 
address the specific problems which ha^ been identified by the Hoover- 
Stanfand Work/Study, teams, the Open Space Work/Study team, Jn "par- 
ticular, found that little has been written of.a general nature which would 
help not ohiy the Hoover teachers in planning their move to a new open- 
yace school, but all . teachers in similar circumstances. Thus, this group 
developed its own manuals, "How to Survive in Open Space," and'"A Guide for 
Teachers Moving into Open Space" which are compilations and distillations of 
everything found in past literature, to which has been added the wisdom gained 
by the team this current year. The Open Space Work/Study team has also 
developed an igstrum^nt Resigned to determine what factors encouraged 
teachers to .woi^k cooperatively, and to enable teachers to monitor anticipated 
problems involved in the move tq an op^sn space school. This instrument has' 
been administered to the Hofaver faculty, and to the faculties of two other Saa^ 
Jose junior high schools approximately, one month before each one moved into 
its new open space school. The instrument will be roadminist^red approximately 
three months after the move to the new schools. 
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: The Valuation Unit has^adJhWiistered questionnaires to the Hoover 
faculty and to the participating Stanford faculty members which will be used 
to assess changes in attitudes along a variety of indicators through the two 
years of the project. 

^ -.The preseryice program uses a pre-post test mode for the cognitive 
aspects of the training provided %y the PDS and the team leader, but the 
interns' experience is not feliant'on research techniques or interventions. 



Target Populations ^ - ^ 

The primary target populations are the Hoover teachers, aides, and 
interns. The students, community, and administrators at Hoover are 
.secondary targets, as are the Stanford participating faculty and research 
assistants. We might call some other .target groups tertiary or peripheral, buf 
the project is being watched by them,\and may have considerable impact on 
Stanford faculty who are not directly involved, and on the administration of 
the San Jose Unified School District. ^ 



Indicators Used To I>etennine Success ; ^ 

Weh ave defined several different kinds of success which we hope to 
achieve. Some invof^ie changing classroom management techniques, such as 
using team-teaching and open space, which are readily observable. Some 
involve making curriculum improvements, which are also easily 
documented To gauge degree of community participation, we use at- 
tehdance records at meetings, lectures, presentations, numbers of volunteers 
for certain tasks, etc. In addition, several surveys (a list follows) have been 
adrhinistered, and we can look at students' grades and scores on standard 
achievement tests. 

' ■ . - 
Language Dorpinance Test (85 seventh .graders) 
Multi-cultural Climate Scales (all Hoover students 

teachers) * ^ ' 

Survey to Hoover teachers 
Survey to participating Stanford faculty 
CTBS scores for al^ Hoover students 




However, a major part of our assessment of success depends on observation 
by project participants and by the evaluation unit. We have come to feel 
that in our particular situation, it is best to keep the number of instruments 
administered and tests given down to an absolute minimum, so that we can 
get full cooperation on the few, carefully chosen ones we deem crucial. ^ 
Because most project participants and the? evaluation unit meet frequently, 
we are able to maintain with a high degree of certainty that our^observations 
are legitimate indicators of success for our proiect. 
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UNIVERSITY OF CENTRAL ARKANSAS PROJECT ' 

Name of Project Director and Evaluatpr 

Dr, Jeff Holland, Director 

P.O. Bol C ' • 

University of Central Arkansas 

Conway, Arkansas 72032 . 

Dr. DarrgM Gentry, Evaluator 
P.O. Bcj^i?> . , 
" University of Central Arkansas 

Conway, Arkansas 72032 - ' 

Description and Listing of Project Goals 

1. To introduce into the teacher-training and retraining 
program, a modeLto increase the proficiency of regular 
classroom . teachers to meet the educational^ needs of 
children with learning or behavioral problems/who are in 
their classrooms. - ' 

2. To provide on-site instruction for regular college students as 
a college supported activity to complement the Retraining 
program. ' ' 

3. To continue the development and implementation, as well as 
the evaluation, of a competency-based teacher education 
prograrti for^ the University of Central Arkansas. < 

4. To provide in-service or site-based retraining activities 
ehnphasizing curriculum ^development and materials 
development among teachers of primary school aged 
children. 

5. To provide parents with home-use materials -to complement 
the school-based educational program and provide training 
in the use of such materials. (To involve parents in the 
learning experiences of their children.) 

6. To become acquaintfedrjwith and participate in the in- 
volvenr>ent of commtjnity resources found in the target 
community. 

Description of Intervention and Treatment 

The Tenth Cycle Teacher Corps Project at the University of Central 
Arkansas has two focal points — improving the prpfessional competency of 
teachers in meeting the special needs of mainstreamed Students, and 
providing selected readiness activities for handicapped students who will be 
malnstrpamed into the regular classroom. 

The paradigm for Improving teacher competency is a competency 
based inservice program developed at the Education Service Center, Region* 
13, Austin, Texas. The materials provide approximately 40-50 hours of in- 
struction delivered in three phases. The program focuses^n skills, concepts, 
and attitudes necessary for elementary teachers to assist mildly handicapped 
students vVith the environmental orientation and adjustment required of 
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them when, they^^re mainstreamed into the regufer /classroom: Mpre 
specifically, the^rogram; deals with the problems of individualizing in- 
struction, utifizinjg alt^natiye management strategies, anii Interfacing the 
regiiJar and sfiecial education teacher. » ,> \^ 

Phase y— Mairistre^ming* — involves approximately 12 Rours of in- 
struction vmich ts facillitatbrTdirected but involves the partrcipants, in* larger 
group.arfcJ small-group activities. Phase II — SkiM Building — 'seeks td 
develop skills through pfebfem-solving activities rehi|e(fHo. individualization/ 
of instruction' Particip^ts work independently or small groups at data 
bajiKS. 6(aftiptes~^of jdata bank topics are: Assessment/Evaluation; .Com- 
municator), Crading/Reporting, Learning Styles, Learning ErwirOnm^ntsj and 
Curriculum %l^ctlon. Phase III — Implementation providl^jhe processes 
and resource?/. for the irhplementatlon of individualized* instifittion. , 

A^a f^s^^ep in the provision of readiness actiyiti^ fdr the special 
education sttraents wha are to be mainitreamed, the' teacher^ interns ad- 
minister an/lnformaj rriulti-disciplinary irtventory. The results at;e used to 
.di^^llkpse'*t]e students' strengths and weakjj^sses. A consliltirig teacher then 
'v>a>rks vAih^^ an individual basis, preparing him or^tter ffer entfV 

.'into'^iv^ |eg^ child is introduced to the next and the 

*^t€;^^di^^}ven Special inMriiction in those sWljs and/;Gonceptjt which have 
'.alre^ief^j qi^ep sti^died by tte* regular stodeh^§rh^/66nsulting leacher also 
. tea^clie'^,^h^uderit the ru the cjassfroonft /*n3;Jntroduce^ or her to 
thV' regular teacher and members of'th^cl4|$, ?i' 

%ar^!j>ppiiUtion .V' '■ . i< \ " • ' \ ' ^ ' 

./frffertt]^^ a fichop|*l|^tj^^ central area of 

liit^e;pb.^W|^ a*5taf,f consisj^iilg %t one priirapaC'^o secretaries, 28 
classrcK3fT)^iJjl|chers, with , fiye special,: tesichersy in^^ two special 

education t^^jSfcfrers, one r^mi^ial t'eacffng teach^r,*ione music teacher, and 
one librariart. thg student body cpnsisjs of 720 students with a ratio of 63% 
black and 37% wHite. One-hi^f^Qr^T^ the students are reading below 
grade levd. TW^nty^one perwiVo^f'Ihe ^tud^nt body is housed, in a Federal 
Housing Project neaf theschoql/T'rtis js theonly neighborhood school in the 
Little Rock School CJistrict. ' V ^^i^^ * 

Description of Success Indica^drs *\- ^ 

One indicator of the success of the Teacher Corps Project should be the 
-impact of the inservlce training upon theteacher. The attitude of the teacher 
toward special education students and toward curriculum will be measured 
using the Curriculum Attitude Inventpry developed by Michael Langenbach, 
University of Oklahoma. This instrument was constructed to discriminate 
between teachers with- positive and those with negative attitudes toward 
curriculum use and planning and has been validated ancf used 'to determine if 
teachers in an inservice situation with curriculum planning exper^ence have a 
mor^ positive attitude toward curriculum use and planning than do inservice 
teachers without such experience. 

Another aspect of the impact upon teachers is knowledge of ex- 
ceptional children and of the placement program for seich children. The 
Riicker-Camble Education Program Scale will be used ^to measure this 
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knowledge. The instrument requires the respondent to suggest the best 
educational placement for 30 different children. Kinds of placement In- 
clude: regular classrooni^ consultant conference^ consultant and direct 
services, resource room, part-time special education classes, full-time special 
education classes, and placement In a residential school rather than the 
public school. ' 

. A' second facet of the project deals with the impact of readiness ac- 
tivities u^Kjn mainstreamed students. Two specific are^s have been-selected 
tfof' measurement - self-concept and academic acKtevement. The Self 
►ervatipn . Scale will be used to measure the way children perceive 
Iselves and their relationships to peers, teacher, and school. It measures 
^'^^j'^^^sjons of self-concept: (1) self acceptance, (2) social maturity, (3) 
.schoW affiliation, (4) self security, and (5) achievement motivation. 



I •^ -r^?.^^'^""^^"^ measure academic achievement will be the SRA 

^^P^^' ^^'^ '5 3 widely-us^d, well-known battery which 
l^r^r^eVeral areas of academic growth. ScoTes from this will be used as 
irt^icator of the progress of mainstreamed students and will also be used 
^i^tornpare tWe performances of regular students who are assigned to classes 
which ' contain mainstreamed students with the performance of regular 
students who are not es^posed to have mainstrieamed students in their 
classroom. It should be noted that the original plan called for the use of the 
Metropolitan Achievement Test to be used for measuring achievement: The 
^^^^"^Sii?.*^^® ^^'^ Achievement Series resulted when the school system 
invoNBiliwUched their system-wid . 
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UNIVERSITY OF OREGON PROJECT . 



Co-directors: Dr. Robert Ham mon 

Dr. Keith Acheson ^^ ' 

Evaluators: 'Meredith Call 
Paul Raff eld 

Address: / Teacher Education division ' 
^ ; College of Education 
' ; University of Orfegon 
^ ; ^ > Eugene, Oregon 97403 

The University of Oregon Teacher Corps Project in Eugefte began with 
bvo fundamental^sumptions. Jhe first is that many inservice teachers are 
jhadequately prepared to meet, the needs of exceptional children. The 
second is that there is an abundance of research-based- teacher training 
material, developed over the past decade with federal funds, which has not 
been compiled and disseminated effectively to inservice teachers. 

Project Goals , 

■ ■ . ■ ' ■ .-I 

A common feature of many of these recently geneipated training 
materialj^.is that they-, are focused toward the exceptional child's needs. 
There^^r^^j an over-riding goal for our project is to facilitate the connection 
of insen^te teachers with research-validated techniques for improving 
instruction of all children, and particularly exceptional children. 

Project Activities 

: .The project research team spent the fall ordering, reviewing and 
evaluating materials from all over the country. If a training package fit into " 
one of our four project categories (diagnostic and prescriptive teaching; 
competency-based curriculum development; program goal |issessment; and 
community resource integration), it was ordered and examin^ rigorously for 
careful research validation data: Of the 255 packages reviewed, only 51 fnet 
our objective criteria for incorporating the best available research. 

Target Population * t. 

At present, project memb^rs^are working closely with teachers, interns,* 
aides, and parents to select from materials which met our criteria and which . 
appear to meet needs in the site schools. 

Our ellmentary site is a Title I designated school where over 50% of the 
student body fits the state definitions of exceptional child. 

The secondary school site also was selected on the'basis of having a 
high concentration of exceptional children.. Clearly we have an aj^propriate 
audience, and the project aim of using the best available traininig^materjals 
to help teajchers teach thes^ students is a critical one. 

The training of our Interns provides another lev^l within this general 
project focus. Oneof our secondary interns - an exceptional person in that * 
sl\e.is blind — is concentrating on reading Instruction with adolescents who 
have not yet:mastered decoding. As she works with her master teacher to 
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I tgain competence with teaching techniques used in her school, she Is 
simultaneously learnihg from project members in university classes about 
research-validated techniques4or teaching reading. Consequently, this iojern 
will be Increasingly able to contribute to^ as well as draw^rom, the expertise 

' available at her training site. ^ • . 

The elementary interns work almost exclusively with ^exceptional, 
children In a variety of settings. There 1s deyelopihg an increasing "give" to 
balance "take" as the ihterns become^aware of what is, verSusKvhat fnight be. 

^ So far we tt&ve found tooperating teachers af both levefs very con- 
cerned about addressing the unique aspects of the exceptionali^hild and 
very receptive tpward new ideas from l^oth the interns and the demon- 
stration componenf of our project. - . 
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WEST VIRGINIA TEACHERX:pRPS PROJECT 
Project Director arid Evaluator ■ ^ 

Director: RonaM B. Childress 

Evaluator: Thomas Ff Breen III, pfogram Pevelop^ment Specialist 

Address^ 224 Hill 'Hall . - 

West Virginia College** of Graduate Studies ! 
n . _ Institute, West Virginia 25112 

Project Goals and Objectives ^ 

A. Community Education * , / ' \ 

1. Develop a horpe^based early childhood education program for 
Clay County parents. , ^ 

2. Facilitate increased corhmuriity participation in 

• educafional/comnrHj.nity programs and projects. 

3. Where feasible,Jevelop a comprehensive community supported 
volunteer tutorial program focused on the expansion of existing 
federally ^funded prbferartis and. on becoming self-supporting 
shoulQl ifederal funding cease. 

B. Research, Documentation, and Evaluation . 

. 1.' Design and establish'a data-based research process-model which 
will provide information on a rural Teacher Training Cpmplex 
W4th respect to participant and total program accountability. 

- ^ 2. Design and establish a management information delivery system 
for the implementation of a rural Teacher Training Complex 
with respect to project aictivities and results. 
3. Desigrt, develop anci operationalize a comprehensive project 
evaluation design. • 

C. Teacher Training and Retraining , • 

' 1. Design and implement a model demonstration 
trjaining/retraining complex in Clay County which will include 
involvement/participation, from the SDE, the local school 
system, the community, and' IHE. ^ 
' 2. Field test an jni|pvatiye craduate level elementary education 
prograrh which will be fiefd-based and responsive to the needs 
of the Appalachian school community which it serves. . 

p. Local Education Agency ^ 

1. Provide opportunities to staff renewal througfi a continuous- 
program of retraining/development. ^ 
- 2, Nurture and enhance the digni%^nd pride of children, teachers 

* and" community * persons In their Appalachian culture and 
^ ^ heritage. 

3. "Design and implement an individually guided* education - 
program in the Teacher Corps Training Complex. 

4, Deterrriine status of curriculum In the Training Complex. 
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Research Intervention • , * 

The principal research obje^ctive of the project is tp test whether the 
College of Graduate Studies Elementary Education program can produce 
significant .measurable changes inMeachers and, consequently, in students. 

. This mode! program represents a rather novel approach to training teachers, 
xdesigned to t^ch them' 'to employ, a diagnostic-prescripftive model of 
t^ching dnd based on the application and adaptation of research models 

^ and findings to the elementary-school classroom. ' 

. Any instructional strategy or technique must be adapted by an in- 
dividual teacher not onjy' to his own personality and stVje but also the 
material which is being taught and to»the individual abilities of the students. 
The Individual teapher is, presently, the only person who can adapt research 
on learning and instruction and on individual differences to the classroom 
setting. 

A non-traditional resfearch-based program has be^n developed at the 
West Virgixiia Cpllege of Q^aduate Studies to train tochers to apply research 
findings in the'^Classroom. This .program calls for a strong fqundation In 
educational research, assessment, curriculum and instructional, planning, 
and evaluation. This strong planning and assessment foundation is coupled 
with some basic, but indepth, study in psychology and human development. 
A complete description of this program may be found in the Elementary 
Education Program Ar^a Self-Stiidy (1975) and in the syllabi of the program 
courses. 

Target Population ' ^ r 

The Elementary Education Graduate program is being offered to the 
interns and to the teachers of the Clay County, West Virginia, school system. 
Specifically, however, the project training complex is located in the Clay 
Elenfientary School, the largest elementary scboql in the county. .The effects'" 
..of the intervention are expected not only on the teachers but also, and a$ a 
consequence, on the Clay Elementary School students. 

Although Clay is a rural Appalachian community, the intervention'^is in 
no way intended to be specific to such a population. The effects should be 
generalizable to almost any group of elementary school teachers and 
students. 

Indicators of Success . 

The project employs basically a pretest/posttest design' to test the 
impact of the graduate program on teachers and students. Among the 
measures are:. ' • 

a. the Minnesota Teacher- Attitude Inventory (MTAI), a measure of 
teacher attitude; \ ^ 

b. the Purdue Teacher Opinionaire, which .measures teacher per-' 
ceptions of the school'ecivlronment, the community; and community 
values; ' 

c. a measure of teacher self-esteem; 

d. a measure of knowledge of content of the graduate program; 

e. the Student Attitude and Activity SurveV, a nieasure 6i student 
attitude toward school and various subjects, student 
self-concept, and participation in individualized learning; ' 
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f. •student achievement, the- Stanford Achievement Tests and the 
\ Educational Deveropment Series; and 

g. teacher classroom behavior, Fjanders Interaction Analysis and,other 
rating scales. ^ 
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